SMB Gateways high memory

[velo]

We have a number of SMB appliances (1530s) running in the network. Everything has been running fine (for the most part) up until around 29/30 May. Suddenly the devices went from around 75% memory to 90%+. This is causing us issues because the devices are crashing and having to be rebooted.

I have looked through the change logs and no major changes happened around that date. One very small push to two firewalls only. 

I'm am running multiple versions with a lot of them being 81.10.10 (but some firewalls still on 80.20.x) So in summary, the issue is not version specific. 

I have opened a case with Checkpoint but it hasn't really provided any answers. They have asked to run a memory script and try catch when a firewall fails but that is easier said than done. 

Has anybody seen similar behaviour? 

Thanks

[Chris_Atkinson]

Are the effected machines all Internet facing (any configured for remote access) or internal ones also?

[velo]

They are all internet facing. None are affected by the latest CVE because they don't have Remote Access VPN enabled. 

Thanks

[the_rock]

Personally, I would try upgrade one of them to latest code and see if any difference. If still same issue, I would open TAC case to check further.

Andy

[velo]

Most are already on the latest code. 

[the_rock]

If thats the case, I would definitely get TAC case going. Just curious, when you say all was fine till end of May, anything happened around that time that could have caused this? Logically, sounds like it could be something else causing this in your network, as it does not make much sense to me it would be smb firmware if it happens on most of devices.

Just my 2 cents...

Andy

[velo]

Thanks, I already have a case open with TAC and it's not really been useful so far. It's still open and we're still looking at it. 

I looked through the logs and there was only one small push on around those dates which was to push a policy to two firewalls. 

I agree on the firmware which is why I mentioned it happening on different firmware versions. 

[the_rock]

K, so couple of points/thoughts.

1) Is it possible to revert those changes to see if it helps?

2) Any way you could upload simple network diagram, so I can try figure out what possible could be causing this?

Best,

Andy

[carl_t]

We are having the same issue on all our SMB appliances, we have had to log a TAC case. we cannot ssh or web into them now causing us all sorts of issues.

It feels a bit of a coincidence since the CVE was released, feels like Checkpoint have "sneaked" an update in somewhere.

none are internet facing.

[the_rock]

I agree with you 100%. Generally in life, I do NOT believe in coincidences...well, it may happen once in a blue moon as they say, but this would certainly seem to be more wide spread issue. Personally, if I were you, I would pick up the phone, call your local SE and tell them about it, see if they can push it further internally.

Andy

[velo]

Wow this is interesting.. Out of interest are you running AV,IPS etc? I had the same suspicions as you. My thoughts were that something was dropped in an IPS or AV update which has caused these issues. 

Funny you mentioned about the SSH and Gaia not working. I have seen that also occasionally. The only way to fix it is to reboot the device. Also sometimes backing up the device via gaia gui crashed the gui. 

Do you have all your devices on monitoring? Are you able to look at historical graphs and see if you're memory spiked around 29/20th? That would be really useful. 

[carl_t]

Hi, all these devices are now saying error on the console, some we can ssh into but it doesn't accept any commands, we can web into others but cant do anything.

We have just upgraded the standby one in a cluster but it wont now fail over.

This is major and Checkpoint need to come forth

[velo]

1) The changes were just pushing the same policy (with no changes) to to SMB devices after I upgraded them so fairly irrelevant I think.

2) It's a very simple topology. Star topology with central firewalls 6000 series and all remote sites being SMB appliances. 

[Chris_Atkinson]

Do you track the volume of connections for these gateways, does it trend with the memory usage or no?

[velo]

Yes I track them. No change in the traffic profile at all (volume of traffic, and number of connections)

[Amir_Ayalon]

Thanks for letting us know.

we are looking into it.

Question - what happened on the  29/30 May ?  policy push ? upgrade ? no change ?

are you running the latest GA ?

[velo]

Morning Amir

It was a simple push after upgrades. I am slowly upgrading all of the SMBs to latest version. I just push the policy after upgrade, that's all it was.

I have got acknowledgement that it's a known issue though so hopefully some traction.

Thanks