High memory usage on 1570/1590

[Morten_O]

Hi,

we have lately hardware-refreshed a lot of 1400-appliance to mainly 1570 and 1590 models.

All are now running R81.10.10 build 996002945

A few times, we have had reports, that the appliances becomes unresponsive (not even answering ping, ssh or webui) and has to be power-cycled to start working again.

So I checked at multiple customers, and I can see, that they are all running with very high memory utilization - above 80%.

All are centrally managed, and I have seen this at multiple customers - so very different policies etc.

One of the customers is not even running IPS, which is known for intense utilization (at least on the 1400-appliances).

Are others seeing the same? Can it be a memoryleak or....?

I already opened a SR (where first recommendation was to upgrade......), but I was just interested in hearing if I'm the only one seeing this picture.

[PhoneBoy]

It is against the EULA to distribute our code outside official channels.
Please be aware of the formal escalation process for TAC here: https://www.checkpoint.com/support-services/check-point-tac-support-escalation-path/ 
Please send me the SR in a PM and I will have a look.

[Amir_Ayalon · Accepted solution]

Hi All

High memory usage on Centrally managed SMB is caused by HCP (healthcheck point) in management server.

HCP tries to run some python commands on SMB appliance which doesn't support Python. Due to this, multiple sfwd instances were created and memory was not released. 

Customers will start to get the automatic HCP update in the next few days. Meanwhile, you can also update the version manually using below steps,

Download HCP TAR

https://support.checkpoint.com/results/download/134058

In Expert mode,

Run:

# autoupdatercli install <Full Path to the TAR Package

run this command to verify hcp version hcp -v

you should see or higher build:

HCP Take: 58
HCP RPM Build: hcp-1-592320.i386
hcp-1-592021.i386

• Reboot the problematic SMB appliance to free the allocated memory. 
• Once the SMB gw is up, push policy again. 

[Marquevis]

I didn't find the .rpm file in the link below:

https://support.checkpoint.com/results/download/134058

Is he really correct?

[Marquevis]

1. Download the HCP TAR

2. Run this command in the Expert mode:

   # autoupdatercli install <Full Path to the TAR Package>

[JeffCote]

I have the same issue on a centrally managed ( Cloud, SMP ) using version 81.10.10(2945).
It affects both a 1600 and a 1570.

[Chris_Atkinson]

Please check that HCP is updated on the management and if the issue persists contact support.

Note Build ending 2993 is the latest and corrects a recent OpenSSH issue.

Edit: SMP managed devices are not affected by the issue described here and will need seperate investigation. 

[JeffCote]

These firewalls are managed through the Spark Management Platform, i'm not sure where i can see the HCP version.

[Tal_Paz-Fridman]

HCP is not supported on Quantum Spark Appliances and does not run on Spark Management Platform (Cloud based solution)

[Morten_O]

It sounds like to root-cause has been found, even though I still haven't seen any of the 15xx's going with less memory-usage, but let's hope this will happen during this week.

I have a question though. HCP is a passive tool, right, which has to be run maunally. The Spark appliances doesn't even support HCP. So how can a HCP on the managementstations cause this issue?

[PhoneBoy]

HCP on the management does monitor gateways (including SMB ones).

[Guru_N_Training]

I know this is old but we have been dealing with the same issue and it is still going on.  This is what we finally got from TAC after many month of tickets and firmware upgrades from R&D:

"The version part itself is not the issue. The amount of memory IPS now requires due to database/protections (example: update in signatures) over time, along with the additional overhead of the other blades is what is causing this memory issue.

Due to this increase in blade memory consumption, we are recommending our sales team reach out to properly size the environment."

[Chris_Atkinson]

Thanks for your update. This would be more helpful if you identify the model of device and firmware version used.

Whilst these smaller devices do have their limits there has been some positive feedback with R81.10.17 and the script from sk183290.

[Guru_N_Training]

Sorry forgot to add that. Same models, 1570 and 1590.  We have 1550s as well but they do not seem to experience the problem of rebooting.

Running R81.10.15 Build 996003913 and in the process of upgrading to R81.10.17 Build 996004508 for our entire SMB environment.

We have tried the script during this process as well.  We are also a Veriti customer so are making use of a lot of IPS changes.

[Timothy_Hall]

May want to give this relatively new 1500 series optimizer script a try, might help: sk183290: Optimizer Script for Quantum Spark Gateways 1500 series.

Ran into this script while researching my upcoming Gaia 4.18 Immersion Course.

[Guru_N_Training]

Yes we were given this script early January when R&D came up with it.