topic Re: memory leak in Firewall and Security Management

memory leak

knassif — Tue, 12 Nov 2024 17:10:07 GMT

we have been having a memory leak issue on our gateway firewalls, we have applied all kinds of patches and hotfixes and we still face the issue, if we leave the memory to get to the max the firewall becomes unresponsive and cant access it via ssh and have to reboot it. appreciate any help or anyone has experienced the same issue attached a screenshot

current version is R81.20 T89

Re: memory leak

the_rock — Tue, 12 Nov 2024 18:13:43 GMT

What do you see from cpview?

Andy

Re: memory leak

the_rock — Tue, 12 Nov 2024 18:15:52 GMT

You can also refer to below.

Andy

https://support.checkpoint.com/results/sk/sk35496

https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/td-p/40528

Re: memory leak

PhoneBoy — Tue, 12 Nov 2024 20:58:02 GMT

You should probably gather the necessary data with: https://support.checkpoint.com/results/sk/sk35496
And engage TAC if you haven't already.

Re: memory leak

knassif — Tue, 17 Dec 2024 19:56:43 GMT

it shows the same thing, once the memory reaches close to the total the firewall stops passing traffic and cant ssh to it have to reboot it screenshot attached

Re: memory leak

knassif — Tue, 12 Nov 2024 21:34:37 GMT

yea we have been troubleshooting this with TAC for I think 3 months with running debugs and updating hotfixes, no luck

Re: memory leak

knassif — Tue, 12 Nov 2024 21:39:19 GMT

we did follow that sk and gave info for TAC like two times no luck still

Re: memory leak

knassif — Tue, 12 Nov 2024 22:15:09 GMT

what i'm seeing is f2f is high

# fwaccel stats -s
Accelerated conns/Total conns : 0/0 (0%)
LightSpeed conns/Total conns : 0/0 (0%)
Accelerated pkts/Total pkts : 0/35313407 (0%)
LightSpeed pkts/Total pkts : 0/35313407 (0%)
F2Fed pkts/Total pkts : 35313407/35313407 (100%)
F2V pkts/Total pkts : 0/35313407 (0%)

Re: memory leak

the_rock — Tue, 12 Nov 2024 22:38:31 GMT

Did you escalate the case? Beccause this sounds pretty serious issue to me...

Andy

Re: memory leak

knassif — Tue, 12 Nov 2024 22:41:15 GMT

yea, any reason why it would be using f2f at 100%? we only have fw as enabled blades.

# enabled_blades

# fwaccel stats -s
Accelerated conns/Total conns : 0/0 (0%)
LightSpeed conns/Total conns : 0/0 (0%)
Accelerated pkts/Total pkts : 0/35313407 (0%)
LightSpeed pkts/Total pkts : 0/35313407 (0%)
F2Fed pkts/Total pkts : 35313407/35313407 (100%)
F2V pkts/Total pkts : 0/35313407 (0%
fw

Re: memory leak

the_rock — Tue, 12 Nov 2024 22:45:39 GMT

I agree. Hey, see if below might be related. I really hope we can help you fix this problem. Having case opened for something this for 3 months has to feel frustrating.

Andy

https://community.checkpoint.com/t5/General-Topics/SecureXL-100-F2Fed-80-30/td-p/95704

https://community.checkpoint.com/t5/General-Topics/Finding-root-cause-for-all-the-F2F-traffic/td-p/51290

Re: memory leak

Chris_Atkinson — Tue, 12 Nov 2024 23:19:05 GMT

What does "fwaccel stat" show by comparison and what model/hardware is this gateway (16200)??

Re: memory leak

knassif — Tue, 12 Nov 2024 23:18:44 GMT

it is a 16k turbo, it is VSX so on VS0 (management VS) it is showing 100% f2f is this normal? on the other VS's I see a variation some are low on f2f some are high over 60%

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
LightSpeed Accel : disabled

Re: memory leak

Chris_Atkinson — Tue, 12 Nov 2024 23:51:01 GMT

Indeed, under normal circumstances you can largely ignore f2f for VS0 and focus on the other VS with the same cmd.(Presumably you don't route traffic for other Virtual Systems via VS0.)

sk32578 talks to the common reasons for f2f traffic further to the hints given by this command where those causes are policy related, how this would relate to a memory leak remains to be seen.

Re: memory leak

the_rock — Wed, 13 Nov 2024 00:47:44 GMT

I agree with @Chris_Atkinson , you can ignore those for VS0, its probably not relevant. Now, if you see if on other VS's, then yes, should be concerned.

Andy

Re: memory leak

knassif — Wed, 13 Nov 2024 13:31:46 GMT

yes on the other VS's we do have f2f over 30%, in one of the VS's it is at 98%

Re: memory leak

knassif — Wed, 13 Nov 2024 13:32:03 GMT

yes on the other VS's we do have f2f over 30%, in one of the VS's it is at 98%

Re: memory leak

Chris_Atkinson — Wed, 13 Nov 2024 13:45:52 GMT

Did you run the fwaccel stat cmd in the context of that VS, what was the output and is the VS active or standby?

Re: memory leak

knassif — Wed, 13 Nov 2024 14:07:11 GMT

here is the output, it is the active VS

# fwaccel stats -s
Accelerated conns/Total conns : 4/4 (100%)
LightSpeed conns/Total conns : 0/4 (0%)
Accelerated pkts/Total pkts : 77545/5273486 (1%)
LightSpeed pkts/Total pkts : 0/5273486 (0%)
F2Fed pkts/Total pkts : 5195941/5273486 (98%)
F2V pkts/Total pkts : 57/5273486 (0%)
CPASXL pkts/Total pkts : 0/5273486 (0%)
PSLXL pkts/Total pkts : 44/5273486 (0%)
CPAS pipeline pkts/Total pkts : 0/5273486 (0%)
PSL pipeline pkts/Total pkts : 0/5273486 (0%)
QOS inbound pkts/Total pkts : 0/5273486 (0%)
QOS outbound pkts/Total pkts : 0/5273486 (0%)
Corrected pkts/Total pkts : 0/5273486 (0%)
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]#
[Expert@idboinfw009:7]# fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+---------------------------------------------------------------------------------+
|0 |KPPAK |enabled | |Acceleration,Cryptography |
| | | | | |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,3DES,DES,AES-128,AES-256,|
| | | | |ESP,LinkSelection,DynamicVPN, |
| | | | |NatTraversal,AES-XCBC,SHA256, |
| | | | |SHA384,SHA512 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
LightSpeed Accel : disabled

Re: memory leak

the_rock — Wed, 13 Nov 2024 14:11:06 GMT

Do other VSs show the same? Just VS0 is different?

Andy