topic Re: SMB Gateways high memory in Spark Firewall (SMB)

SMB Gateways high memory

velo — Mon, 17 Jun 2024 14:59:08 GMT

We have a number of SMB appliances (1530s) running in the network. Everything has been running fine (for the most part) up until around 29/30 May. Suddenly the devices went from around 75% memory to 90%+. This is causing us issues because the devices are crashing and having to be rebooted.

I have looked through the change logs and no major changes happened around that date. One very small push to two firewalls only.

I'm am running multiple versions with a lot of them being 81.10.10 (but some firewalls still on 80.20.x) So in summary, the issue is not version specific.

I have opened a case with Checkpoint but it hasn't really provided any answers. They have asked to run a memory script and try catch when a firewall fails but that is easier said than done.

Has anybody seen similar behaviour?

Thanks

Re: SMB Gateways high memory

Chris_Atkinson — Mon, 17 Jun 2024 11:46:26 GMT

Are the effected machines all Internet facing (any configured for remote access) or internal ones also?

Re: SMB Gateways high memory

velo — Mon, 17 Jun 2024 12:06:24 GMT

They are all internet facing. None are affected by the latest CVE because they don't have Remote Access VPN enabled.

Thanks

Re: SMB Gateways high memory

the_rock — Mon, 17 Jun 2024 12:59:44 GMT

Personally, I would try upgrade one of them to latest code and see if any difference. If still same issue, I would open TAC case to check further.

Andy

Re: SMB Gateways high memory

velo — Mon, 17 Jun 2024 13:02:28 GMT

Most are already on the latest code.

Re: SMB Gateways high memory

the_rock — Mon, 17 Jun 2024 13:07:12 GMT

If thats the case, I would definitely get TAC case going. Just curious, when you say all was fine till end of May, anything happened around that time that could have caused this? Logically, sounds like it could be something else causing this in your network, as it does not make much sense to me it would be smb firmware if it happens on most of devices.

Just my 2 cents...

Andy

Re: SMB Gateways high memory

velo — Mon, 17 Jun 2024 13:10:21 GMT

Thanks, I already have a case open with TAC and it's not really been useful so far. It's still open and we're still looking at it.

I looked through the logs and there was only one small push on around those dates which was to push a policy to two firewalls.

I agree on the firmware which is why I mentioned it happening on different firmware versions.

Re: SMB Gateways high memory

the_rock — Mon, 17 Jun 2024 13:12:03 GMT

K, so couple of points/thoughts.

1) Is it possible to revert those changes to see if it helps?

2) Any way you could upload simple network diagram, so I can try figure out what possible could be causing this?

Best,

Andy

Re: SMB Gateways high memory

Chris_Atkinson — Mon, 17 Jun 2024 13:32:28 GMT

Do you track the volume of connections for these gateways, does it trend with the memory usage or no?

Re: SMB Gateways high memory

carl_t — Mon, 17 Jun 2024 13:33:43 GMT

We are having the same issue on all our SMB appliances, we have had to log a TAC case. we cannot ssh or web into them now causing us all sorts of issues.

It feels a bit of a coincidence since the CVE was released, feels like Checkpoint have "sneaked" an update in somewhere.

none are internet facing.

Re: SMB Gateways high memory

the_rock — Mon, 17 Jun 2024 13:37:33 GMT

I agree with you 100%. Generally in life, I do NOT believe in coincidences...well, it may happen once in a blue moon as they say, but this would certainly seem to be more wide spread issue. Personally, if I were you, I would pick up the phone, call your local SE and tell them about it, see if they can push it further internally.

Andy

Re: SMB Gateways high memory

velo — Mon, 17 Jun 2024 13:47:55 GMT

Wow this is interesting.. Out of interest are you running AV,IPS etc? I had the same suspicions as you. My thoughts were that something was dropped in an IPS or AV update which has caused these issues.

Funny you mentioned about the SSH and Gaia not working. I have seen that also occasionally. The only way to fix it is to reboot the device. Also sometimes backing up the device via gaia gui crashed the gui.

Do you have all your devices on monitoring? Are you able to look at historical graphs and see if you're memory spiked around 29/20th? That would be really useful.

Re: SMB Gateways high memory

velo — Mon, 17 Jun 2024 13:50:09 GMT

Yes I track them. No change in the traffic profile at all (volume of traffic, and number of connections)

Re: SMB Gateways high memory

carl_t — Mon, 17 Jun 2024 14:08:09 GMT

Hi, all these devices are now saying error on the console, some we can ssh into but it doesn't accept any commands, we can web into others but cant do anything.

We have just upgraded the standby one in a cluster but it wont now fail over.

This is major and Checkpoint need to come forth

Re: SMB Gateways high memory

velo — Mon, 17 Jun 2024 14:26:08 GMT

1) The changes were just pushing the same policy (with no changes) to to SMB devices after I upgraded them so fairly irrelevant I think.

2) It's a very simple topology. Star topology with central firewalls 6000 series and all remote sites being SMB appliances.

Re: SMB Gateways high memory

Amir_Ayalon — Tue, 18 Jun 2024 08:18:49 GMT

Thanks for letting us know.

we are looking into it.

Question - what happened on the 29/30 May ? policy push ? upgrade ? no change ?

are you running the latest GA ?

Re: SMB Gateways high memory

velo — Tue, 18 Jun 2024 08:29:21 GMT

Morning Amir

It was a simple push after upgrades. I am slowly upgrading all of the SMBs to latest version. I just push the policy after upgrade, that's all it was.

I have got acknowledgement that it's a known issue though so hopefully some traction.

Thanks

Re: SMB Gateways high memory

lrossi89 — Thu, 27 Jun 2024 09:20:57 GMT

Hi Velo,

did you solve the problem ?

We have the same problem (high ram from late May) 😞

Re: SMB Gateways high memory

Chris_Atkinson — Thu, 27 Jun 2024 10:05:54 GMT

You may also wish to review this discussion:

https://community.checkpoint.com/t5/SMB-Gateways-Spark/High-memory-usage-on-1570-1590/m-p/217894#M10818

Re: SMB Gateways high memory

velo — Thu, 27 Jun 2024 10:36:24 GMT

Thanks Chris. I have been updated via the open case. An interim fix has been provided. Alternatively this will be fixed next week in the HCP script update. Thanks for the info.