topic Re: SMEM's "Failed Allocation" value is very high. in Firewall and Security Management

SMEM's "Failed Allocation" value is very high.

Young_Wook_Choi — Tue, 15 Dec 2020 10:44:20 GMT

Hello,

All traffic going through the firewall was cut off.
The firewall is using the IPS blade function with the FW blade. (R80.10)
CPU usage is not very high. However, the memory usage seems very high.

The output of the fw ctl pstat command is shown below.

This firewall has an uptime of 3 weeks.
SMEM's "Failed Allocation" value is very high. (It is more than about 4,600,000)
This count was recorded for only 3 weeks.

This system has 16GB of memory (RAM).

When looking at the CPVIEW output, memory usage is always high as shown below.

Could a session be disconnected due to high memory usage?

I opened a Case with this problem, but TAC doesn't help. Rather, they say the system is normal.

Will upgrading this system's physical memory solve this problem?

Over the past few weeks, we've been experiencing a problem of disconnecting service sessions several times.

Re: SMEM's "Failed Allocation" value is very high.

Young_Wook_Choi — Tue, 15 Dec 2020 13:02:03 GMT

Aggressive Aging feature is active. (Default)
Is there a possibility that the service session is disconnected by this feature? (If you guess from the above memory usage)

If the gateway has ever had this function active, where can I check it? (/var/log/messages? or fwd.elg?)

Could this issue be related to this feature?

Re: SMEM's "Failed Allocation" value is very high.

Timothy_Hall — Wed, 16 Dec 2020 15:50:11 GMT

Depending on the load, 12GB of memory being used by the kernel of of 16GB total may be high. Could indicate a memory leak, would recommend ensuring you have the latest GA R80.10 Jumbo HFA loaded for starters as there are almost certainly fixes for memory leaks contained within it. Please provide the full output of fw ctl pstat as that will provide connection statistics and sync info as well. It doesn't look like your firewall is dipping very far into swap space, but please provide output of free -m to see more detailed memory usage and help determine if a RAM upgrade is needed.

Once a connection is up and running, I think all necessary memory allocations for state table space have been performed so I don't believe an existing connection could be killed by a memory shortage, but new ones could certainly be prevented from starting which depending on how an application might be utilizing multiple connections could look like an existing connection getting killed.

Re: SMEM's "Failed Allocation" value is very high.

Young_Wook_Choi — Thu, 17 Dec 2020 04:50:36 GMT

Thanks for your reply.
TAC provided us with a script that could detect memory leaks. And they said this issue was not a memory leak.

Re: SMEM's "Failed Allocation" value is very high.

Timothy_Hall — Thu, 17 Dec 2020 14:23:20 GMT

You have a pretty large number of concurrent connections for a system with only 16GB of RAM (176k current/313k peak); it looks like the kernel is using almost all the RAM (and trying to get more) and the firewall's processes are having to dip into swap space to the tune of 1.3GB.

Assuming a memory leak is not present it does look like a RAM upgrade to 32GB will help here. Perhaps a later version of code will utilize kernel memory a bit more efficiently, but it looks like you just have a very busy firewall there that could use more memory.

Re: SMEM's "Failed Allocation" value is very high.

Young_Wook_Choi — Thu, 17 Dec 2020 15:12:52 GMT

Thank you for your advice. We are planning an upgrade to 64gb of RAM. Also, an upgrade to R80.40 is planned. We hope this plan helps us.

Re: SMEM's "Failed Allocation" value is very high.

Benedikt_Weissl — Thu, 17 Dec 2020 18:16:49 GMT

https://community.checkpoint.com/t5/General-Topics/R80-x-Performance-Tuning-Tip-Connection-Table/td-p/41581 See Tip 4 and Tip 7 for advice on how to lower session lifetime, this could also reduce the size of the connection table. Please use caution when adjusting session lifetime.