Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
velo
Contributor

SMB Gateways high memory

We have a number of SMB appliances (1530s) running in the network. Everything has been running fine (for the most part) up until around 29/30 May. Suddenly the devices went from around 75% memory to 90%+. This is causing us issues because the devices are crashing and having to be rebooted.

I have looked through the change logs and no major changes happened around that date. One very small push to two firewalls only. 

I'm am running multiple versions with a lot of them being 81.10.10 (but some firewalls still on 80.20.x) So in summary, the issue is not version specific. 

I have opened a case with Checkpoint but it hasn't really provided any answers. They have asked to run a memory script and try catch when a firewall fails but that is easier said than done. 

Has anybody seen similar behaviour? 

 

memory-safe.png

 

Thanks

0 Kudos
20 Replies
Chris_Atkinson
Employee Employee
Employee

Are the effected machines all Internet facing (any configured for remote access) or internal ones also?

CCSM R77/R80/ELITE
0 Kudos
velo
Contributor

They are all internet facing. None are affected by the latest CVE because they don't have Remote Access VPN enabled. 

Thanks

0 Kudos
the_rock
Legend
Legend

Personally, I would try upgrade one of them to latest code and see if any difference. If still same issue, I would open TAC case to check further.

Andy

0 Kudos
velo
Contributor

Most are already on the latest code. 

the_rock
Legend
Legend

If thats the case, I would definitely get TAC case going. Just curious, when you say all was fine till end of May, anything happened around that time that could have caused this? Logically, sounds like it could be something else causing this in your network, as it does not make much sense to me it would be smb firmware if it happens on most of devices.

Just my 2 cents...

Andy

0 Kudos
velo
Contributor

Thanks, I already have a case open with TAC and it's not really been useful so far. It's still open and we're still looking at it. 

I looked through the logs and there was only one small push on around those dates which was to push a policy to two firewalls. 

I agree on the firmware which is why I mentioned it happening on different firmware versions. 

0 Kudos
the_rock
Legend
Legend

K, so couple of points/thoughts.

1) Is it possible to revert those changes to see if it helps?

2) Any way you could upload simple network diagram, so I can try figure out what possible could be causing this?

Best,

Andy

0 Kudos
carl_t
Contributor

We are having the same issue on all our SMB appliances, we have had to log a TAC case. we cannot ssh or web into them now causing us all sorts of issues.

It feels a bit of a coincidence since the CVE was released, feels like Checkpoint have "sneaked" an update in somewhere.

none are internet facing.

0 Kudos
(1)
the_rock
Legend
Legend

I agree with you 100%. Generally in life, I do NOT believe in coincidences...well, it may happen once in a blue moon as they say, but this would certainly seem to be more wide spread issue. Personally, if I were you, I would pick up the phone, call your local SE and tell them about it, see if they can push it further internally.

Andy

0 Kudos
velo
Contributor

Wow this is interesting.. Out of interest are you running AV,IPS etc? I had the same suspicions as you. My thoughts were that something was dropped in an IPS or AV update which has caused these issues. 

Funny you mentioned about the SSH and Gaia not working. I have seen that also occasionally. The only way to fix it is to reboot the device. Also sometimes backing up the device via gaia gui crashed the gui. 

Do you have all your devices on monitoring? Are you able to look at historical graphs and see if you're memory spiked around 29/20th? That would be really useful. 

 

carl_t
Contributor

Hi, all these devices are now saying error on the console, some we can ssh into but it doesn't accept any commands, we can web into others but cant do anything.

We have just upgraded the standby one in a cluster but it wont now fail over.

This is major and Checkpoint need to come forth

(1)
velo
Contributor

1) The changes were just pushing the same policy (with no changes) to to SMB devices after I upgraded them so fairly irrelevant I think.

2) It's a very simple topology. Star topology with central firewalls 6000 series and all remote sites being SMB appliances. 

Chris_Atkinson
Employee Employee
Employee

Do you track the volume of connections for these gateways, does it trend with the memory usage or no?

CCSM R77/R80/ELITE
0 Kudos
velo
Contributor

Yes I track them. No change in the traffic profile at all (volume of traffic, and number of connections)

Amir_Ayalon
Employee
Employee

Thanks for letting us know.

we are looking into it.

Question - what happened on the  29/30 May ?  policy push ? upgrade ? no change ?

are you running the latest GA ?

(1)
velo
Contributor

Morning Amir

It was a simple push after upgrades. I am slowly upgrading all of the SMBs to latest version. I just push the policy after upgrade, that's all it was.

I have got acknowledgement that it's a known issue though so hopefully some traction.

 

Thanks

lrossi89
Contributor

Hi Velo,

did you solve the problem ?

We have the same problem (high ram from late May) 😞

0 Kudos
Chris_Atkinson
Employee Employee
Employee

0 Kudos
(1)
velo
Contributor

Thanks Chris. I have been updated via the open case. An interim fix has been provided. Alternatively this will be fixed next week in the HCP script update. Thanks for the info.

the_rock
Legend
Legend

Good to know!

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events