Quantum Spark 1800 high memory usage

[Tim2]

Hi.

I want to ask about high memory usage.

I'm using Quantum Spark 1800 Appliance, Version: R81.10.08 (996001608) .

With all Threat Protections disabled and around 2000 connections, the device RAM usage is at approx 70%.

With enabled Threat Protections and around 2000 connections, the device RAM usage is around 80%.

RAM usage is calculated from the output of free command.

During our "normal" usage, we have approx 10 thousand active connections (the peak is 12000) the device starts to drop connections.

Is it normal that the device has approx 80% of RAM used with only 2000 active  connections?

The top command shows that cpviewd takes 1940m of virtual memory. Is that normal?

Here is output of free:

[]# free -m
total used free shared buff/cache available
Mem: 7903468 5946056 499472 97088 1457940 1669240
Swap: 0 0 0

Thanks.
Tim

[_Val_]

The OS reading of free memory is inaccurate at best. All you need to look at it that your appliance does not swap.

[KristofV]

Hello VAL,

Unfortunately, this inaccurate value is shown in Dashboard and this is what customers see, a red value for memory. I understand this is not the actual value for memory, but difficult to explain to customers !

[Tim2]

Let me add: We are reading the memory status by SNMP record memFreeReal64 (OID 1.3.6.1.4.1.2620.1.6.7.4.5).

There is also an SNMP trap for "High memory utilization" (OID 1.3.6.1.4.1.2620.1.2000.4.2) that alets on high memory usage with default setting of >= 80% of used memory.

IPS has also a dedicated "bypass under load" configuration. So learning about high memory is IMHO important.

So my questions are:

1. What is a good way to lean about the "actual" free memory?

2. Is is normal that the cpviewd process takes 1940m of virtual memory?

[Chris_Atkinson]

Note 996001683 is the current build available for R81.10.08 should you wish to try it. 

[Tim2]

Does this FW update address any memory issues?

[adamec]

Does this FW update address any memory issues?

[Chris_Atkinson]

TAC will be able to confirm this more readily than I but will see what I can find for you and revert.

Meanwhile please note R81.10.10 is the current release with additional fixes & features - refer: sk181134

[PhoneBoy]

According to the free -m output, you’ve got 1.6 GB available…which seems ok to me.
Yes, some of that memory is allocated for other purposes (1.4 GB of memory), but it can be made available if required.

[adamec]

Hi, so is it okay to have 80% memory usage with 2000 connections on average? No reason to be worried?

How can the memory can be made available if required? 

Thanks in advance

[PhoneBoy]

The underlying Linux kernel manages all this.
The main thing to keep an eye on in "free" output is "available" which can fluctuate based on usage/traffic.
Absent any other symptoms, this is normal/expected behavior. 

[KristofV]

'cpview' provides a detailed breakdown of memory consumption, including unused portions of reserved memory. Incorporating the unused portion of reserved memory results in a significantly lower memory usage figure. Unfortunately, Check Point's dashboard doesn't reflect this adjusted memory figure, causing confusion for customers who may see a high memory consumption alert and inquire with their partners.

[Lesley]

I don't see how this post is relevant for the topic that has been started by Tim. The symptoms you describe are not related to the current topic. 

[Tim2]

Hi KristofV.

Thanks for your reply.

Cpview shows me in the Overview -> Memory section the following:

Name, Total MB, Used MB, Free MB

Physical, 7718, 7202, 515

FW, 5711,1559,4152

Swap 0,0,0

In advanced -> Memory -> Overview, it shows the following "Firewall memory usage summary"

Total - 5711MB

Used - 1558MB

Usage - 27%

Free - 4152MB

Does it mean that the device has reserved a lot of memory for FW, but sill has 4152MB free for FW functionality?

Regards, Tim

[Lesley · Accepted solution]

Check this post:

After Briefly looking into it (we are still looking) we found that the memory reserved for some processes was increased (double) than in previous build. it's not a real issue as the memory is just reserved not allocated, and can be freed if needed. it does however change the amount of memory presented as "Free".

This will explain part of the "lost memory" you see. 

Stated in: https://community.checkpoint.com/t5/SMB-Gateways-Spark/Memory-increase-in-FW-s-model-SMB-1800-after-...

[KristofV]

Hello Tim2,

Indeed, that's accurate according to me. It indicates that the firewall has allocated a significant portion of memory for the FW, with 27% of this allocation currently utilized.

[Lesley]

As Val stated you should check SWAP, this is now 0 so no memory related issues.

Linux eats up all the memory it wants. If some memory is taken it does not mean it is really used.

Connection drops after 12k how is this found? Do you see error logs or?

Please share output:

fw tab -t connections -s

fw tab -t connections | grep limit

[Tim2]

Hi Lesley.

The connections drops were our observation.

Here are the outputs:

# fw tab -t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 1453 16543 4019

# fw tab -t connections | grep limit
dynamic, id 8158, num ents 717, load factor 0.1, attributes: keep, sync, aggressive aging, kbufs 21 22 23 24 25 26 27 28 29 30 31 32 33 34, expires 25, refresh, , hashsize 65536, limit 150000

Thanks.

[Lesley]

You peak 16543 with connections and limit is set to 150000. The higher the limit the more memory it uses. It should be not to much. You can try to lower the value or change to 'automatic'. Also the peak is the highest moment after the last reboot. So if you have a fresh uptime the peak could change later.

Second I see they fix this in R81.10.10, you can consider to install it.

SMBGWY-6869

Core

On 1600/1800 Quantum Spark appliances, the size of the Firewall memory pool's initial allocation (hmem) is too high. As a result, the available memory is lower than expected.

[KristofV]

I initially assumed that adjusting the connection limit would affect the reserved memory, but my tests proved otherwise.

[yahavb]

Hi, this is not answering the original question in this thread but I wanted to share here about the visibility enhancements made in Spark Management application, allowing to monitor the system resources and network connection over time, including the memory usage and connections.

It is supported for gateways running R81.10.10 and above, connected to Spark Management application in Infinity Portal.
Some screen shots below.
Feel free to contact me by PM or email for questions an more information.

Thanks, Yahav.

Gateway system monitoring view in Spark ManagementGateway internet monitoring view in Spark Management