memory leak

knassif

we have been having a memory leak issue on our gateway firewalls, we have applied all kinds of patches and hotfixes and we still face the issue, if we leave the memory to get to the max the firewall becomes unresponsive and cant access it via ssh and have to reboot it. appreciate any help or anyone has experienced the same issue attached a screenshot

current version is R81.20 T89

the_rock

What do you see from cpview?

Andy

knassif

it shows the same thing, once the memory reaches close to the total the firewall stops passing traffic and cant ssh to it have to reboot it screenshot attached

the_rock

You can also refer to below.

Andy

https://support.checkpoint.com/results/sk/sk35496

https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/td-p/4...

knassif

what i'm seeing is f2f is high

# fwaccel stats -s
Accelerated conns/Total conns : 0/0 (0%)
LightSpeed conns/Total conns : 0/0 (0%)
Accelerated pkts/Total pkts : 0/35313407 (0%)
LightSpeed pkts/Total pkts : 0/35313407 (0%)
F2Fed pkts/Total pkts : 35313407/35313407 (100%)
F2V pkts/Total pkts : 0/35313407 (0%)

PhoneBoy

You should probably gather the necessary data with: https://support.checkpoint.com/results/sk/sk35496
And engage TAC if you haven't already.

knassif

yea we have been troubleshooting this with TAC for I think 3 months with running debugs and updating hotfixes, no luck

knassif

we did follow that sk and gave info for TAC like two times no luck still

the_rock

Did you escalate the case? Beccause this sounds pretty serious issue to me...

Andy

knassif

yea, any reason why it would be using f2f at 100%? we only have fw as enabled blades.

# enabled_blades

# fwaccel stats -s
Accelerated conns/Total conns : 0/0 (0%)
LightSpeed conns/Total conns : 0/0 (0%)
Accelerated pkts/Total pkts : 0/35313407 (0%)
LightSpeed pkts/Total pkts : 0/35313407 (0%)
F2Fed pkts/Total pkts : 35313407/35313407 (100%)
F2V pkts/Total pkts : 0/35313407 (0%
fw

the_rock

I agree. Hey, see if below might be related. I really hope we can help you fix this problem. Having case opened for something this for 3 months has to feel frustrating.

Andy

https://community.checkpoint.com/t5/General-Topics/SecureXL-100-F2Fed-80-30/td-p/95704

https://community.checkpoint.com/t5/General-Topics/Finding-root-cause-for-all-the-F2F-traffic/td-p/5...

Chris_Atkinson

What does "fwaccel stat" show by comparison and what model/hardware is this gateway (16200)??

CCSM R77/R80/ELITE

knassif

it is a 16k turbo, it is VSX so on VS0 (management VS) it is showing 100% f2f is this normal? on the other VS's I see a variation some are low on f2f some are high over 60%

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
LightSpeed Accel : disabled

Chris_Atkinson

Indeed, under normal circumstances you can largely ignore f2f for VS0 and focus on the other VS with the same cmd.(Presumably you don't route traffic for other Virtual Systems via VS0.)

sk32578 talks to the common reasons for f2f traffic further to the hints given by this command where those causes are policy related, how this would relate to a memory leak remains to be seen.

CCSM R77/R80/ELITE

knassif

yes on the other VS's we do have f2f over 30%, in one of the VS's it is at 98%

the_rock

I agree with @Chris_Atkinson , you can ignore those for VS0, its probably not relevant. Now, if you see if on other VS's, then yes, should be concerned.

Andy

knassif

yes on the other VS's we do have f2f over 30%, in one of the VS's it is at 98%

Chris_Atkinson

Did you run the fwaccel stat cmd in the context of that VS, what was the output and is the VS active or standby?

CCSM R77/R80/ELITE

knassif

here is the output, it is the active VS

# fwaccel stats -s
Accelerated conns/Total conns : 4/4 (100%)
LightSpeed conns/Total conns : 0/4 (0%)
Accelerated pkts/Total pkts : 77545/5273486 (1%)
LightSpeed pkts/Total pkts : 0/5273486 (0%)
F2Fed pkts/Total pkts : 5195941/5273486 (98%)
F2V pkts/Total pkts : 57/5273486 (0%)
CPASXL pkts/Total pkts : 0/5273486 (0%)
PSLXL pkts/Total pkts : 44/5273486 (0%)
CPAS pipeline pkts/Total pkts : 0/5273486 (0%)
PSL pipeline pkts/Total pkts : 0/5273486 (0%)
QOS inbound pkts/Total pkts : 0/5273486 (0%)
QOS outbound pkts/Total pkts : 0/5273486 (0%)
Corrected pkts/Total pkts : 0/5273486 (0%)
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]# fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+---------------------------------------------------------------------------------+
|0 |KPPAK |enabled | |Acceleration,Cryptography |
| | | | | |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,3DES,DES,AES-128,AES-256,|
| | | | |ESP,LinkSelection,DynamicVPN, |
| | | | |NatTraversal,AES-XCBC,SHA256, |
| | | | |SHA384,SHA512 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
LightSpeed Accel : disabled

the_rock

Do other VSs show the same? Just VS0 is different?

Andy

knassif

other VS's some of them are showing very high on the f2f others are operating in below 30% here is one of the outputs below of one of the VS's where we have traffic

here is the output, it is the active VS

# fwaccel stats -s
Accelerated conns/Total conns : 4/4 (100%)
LightSpeed conns/Total conns : 0/4 (0%)
Accelerated pkts/Total pkts : 77545/5273486 (1%)
LightSpeed pkts/Total pkts : 0/5273486 (0%)
F2Fed pkts/Total pkts : 5195941/5273486 (98%)
F2V pkts/Total pkts : 57/5273486 (0%)
CPASXL pkts/Total pkts : 0/5273486 (0%)
PSLXL pkts/Total pkts : 44/5273486 (0%)
CPAS pipeline pkts/Total pkts : 0/5273486 (0%)
PSL pipeline pkts/Total pkts : 0/5273486 (0%)
QOS inbound pkts/Total pkts : 0/5273486 (0%)
QOS outbound pkts/Total pkts : 0/5273486 (0%)
Corrected pkts/Total pkts : 0/5273486 (0%)
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]# fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+---------------------------------------------------------------------------------+
|0 |KPPAK |enabled | |Acceleration,Cryptography |
| | | | | |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,3DES,DES,AES-128,AES-256,|
| | | | |ESP,LinkSelection,DynamicVPN, |
| | | | |NatTraversal,AES-XCBC,SHA256, |
| | | | |SHA384,SHA512 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
LightSpeed Accel : disabled

the_rock

Ok, if its not too much to ask here and if you are allowed to post it, can you list things done so far in TAC case?

Andy

knassif

not sure if because those VS's have not a lot of traffic the firewall is just choosing to go via f2f, what we did in TAC we followed that SK for memory leak that I attached 'memleak3.png' and we upgraded the firewalls to the latest takes and ran the sk memory leak procedure again and we still face the issue, we are on R81.20 T89 currently.

the_rock

Hey, quick question...are you able/allowed to send me the debugs you did for TAC? Im more than happy to review them myself and see if I can assist. If yes, please be free to message me offline and we can connect.

Best,

Andy

Ilya_Yusupov

Hi @knassif,

can you please share with me the tac case number? i will review and will try to assist.

Thanks,

Ilya

the_rock

@knassif I can tell you and I feel very good about this, as @Ilya_Yusupov helped me before with ISPR issue for a customer, he is amazing and will always follow up until issue is solved. You are 100% in good hands mate.

Andy

knassif

the case # is SR#6-0004045311

the_rock

I have no ounce of doubt that @Ilya_Yusupov will help you fix this issue. I dealt with him before and you can tell he truly CARES, 100%. He is a good person.

Andy

knassif

yes give some time i'll upload those

the_rock

No rush, take your time. As soon as you send them, I will download and review. I may message you directly for some details beforehand.

Andy

Are you a member of CheckMates?

memory leak