High memory usage

[abihsot__]

Hello,

Wanted to share the issue we have with our gateway.  We have following blades enabled:

fw urlf appi identityServer SSL_INSPECT content_awareness mon

Appliance is with 16gb, running latest R80.30.

The problem we are having is that at some point memory usage increases sharply and it never comes down, unless we reboot appliance. This is causing issues to the traffic because some connections are getting disconnected during occurrence. I can't find in top (shift+m) any process which would contribute to this behaviour.

I hope I am not alone with this issue, so please give a shout if you have something similar. Some of the occurrences from the past to show what happens:

[Kaloyan_Kirchev]

Sorry but there is no such command neither in bin/bash nor in /etc/cli.sh 🙂

[HristoGrigorov]

What about 'cat /proc/meminfo' ?

[Kaloyan_Kirchev]

Here it is? Any ideas?

[Daniel_Schlifka]

Hmm, i believed all R80.40 would use the Dynamic Clish(sk144112) already, my mistake.
In this case the command is "fw ctl pstat".

[HristoGrigorov]

Notice that almost all available memory is in Active state which means it is constantly being pulled in for use as cache or buffers. It can and it will be used by the kernel if it runs short on process memory but only if very necessary.

How is the system performance otherwise? What 'top | head -n 10' shows ?

[Kaloyan_Kirchev]

As I mentioned, this java keep using memory.

[emreturkmenler]

Hi,

We have a similar issue but not sure if others have the same outputs.

Running R80.30 Take 155.

On top command I see high usage of memory as well as 'free -m' output;

top - 08:52:30 up 26 days, 21:03, 5 users, load average: 1.79, 2.20, 2.19
Tasks: 342 total, 3 running, 339 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 0.6%sy, 0.3%ni, 91.1%id, 0.0%wa, 0.1%hi, 7.3%si, 0.0%st
Mem: 32784868k total, 31227084k used, 1557784k free, 578016k buffers
Swap: 33551744k total, 196k used, 33551548k free, 20191296k cached

# free -m
total used free shared buffers cached
Mem: 32016 30494 1521 0 564 19718
-/+ buffers/cache: 10212 21804
Swap: 32765 0 32765

On the other hand when I check from the cpview I see only %17 of usage , also Smart Console shows low memory usage.

# fw ctl pstat

System Capacity Summary:
Memory used: 17% (4098 MB out of 24012 MB) - below watermark
Concurrent Connections: 11179 (Unlimited)
Aggressive Aging is enabled, not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 4279132160 bytes in 1044710 (4096 bytes) blocks using 2 pools
Initial memory allocated: 2516582400 bytes (Hash memory extended by 1762549760 bytes)
Memory allocation limit: 20142096384 bytes using 512 pools
Total memory bytes used: 0 unused: 4279132160 (100.00%) peak: 3975273208
Total memory blocks used: 0 unused: 1044710 (100%) peak: 1024447
Allocations: 405809094 alloc, 0 failed alloc, 390362140 free

System kernel memory (smem) statistics:
Total memory bytes used: 5775472876 peak: 6295815528
Total memory bytes wasted: 44266902
Blocking memory bytes used: 34193880 peak: 39109936
Non-Blocking memory bytes used: 5741278996 peak: 6256705592
Allocations: 146815898 alloc, 0 failed alloc, 146791213 free, 0 failed free
vmalloc bytes used: 5706239208 expensive: no

Kernel memory (kmem) statistics:
Total memory bytes used: 2973191124 peak: 5738297476
Allocations: 552592884 alloc, 0 failed alloc
537127921 free, 0 failed free
External Allocations: 1991784 for packets, 30309726 for SXL

[abihsot__]

Hi,

its fine, don't worry. Most of your memory is cached and is ready to be used if needed. If you really have memory issues you should see high peak values in Advanced->memory->overview. Also when memory is fully consumed you should start getting failure to allocate messages in advanced->memory->smem-failures.

Check my screenshots in previous posts.

By the way, I heard that checkpoint has fixed some bad memory leaks in R80.30 JHF16x. We are running JHF155 and looking forward to JHF191, so hopefully we won't have those nasty issues anymore...

[emreturkmenler]

Thanks for the Reply,

I did check advanced->memory->smem-failures. but didn't see anything while there were complaints yesterday.Still don't see anything on that screen.

We have 2 clusters running r80.30 T155 and both are the same and was planning to do a new upgrade but bit curious to do it after this even though it may look normal.

Why is the cpview different , does cpview exclude the cashed memory as also snmp outputs on our monitoring tool is getting the data from the cpview as it doesn't alert us.

[abihsot__]

If there were complains, why would you think it was memory related?

You are running 32GB system and kmem peak was ~5GB. You had plenty of memory.

Memory is a complex thing having many parameters. You should compare apples to apples, so double-check what you were monitoring in your tool.

[emreturkmenler]

People were complaining of connection issues which I believe it will be user connection problem but was curious when digging in.

One last thing, on the top command there is a swap output which looks it is being used different than the 'free -m' output. Is this normal?because I don't see this usage on the firewalls running R80.10.

[abihsot__]

Yes, I have it too on some gateways. Difference comes from here.

# cat /proc/meminfo |grep -i swap
SwapCached: 0 kB
SwapTotal: 18836204 kB
SwapFree: 18836072 kB

Mem: 16230576k total, 14246328k used, 1984248k free, 333276k buffers
Swap: 18836204k total, 132k used, 18836072k free, 5901604k cached

Maybe there is a way to find out which process allocated that swap space, but really you shouldn't worry about 200Kb.

[emreturkmenler]

You're correct , thanks for the replies.
I prefer checking the memory usage from the cpview values as our monitoring and also the Smart console dashboard is also showing these values.

[makin]

Do you still have the problem ? 

[emreturkmenler]

This was long time ago and don't have it anymore as we've moved on to newer versions.

[CheckPointerXL]

memory problems are timeless and versionless 🙂

anyway this topic is one of the most easy to reach when you googling about memory on check point, so it could easy picks up a lot of information for a lot of people who needs help

[CheckPointerXL]

Hi all, anyone can help me to correctly verify if this behavoir is normal?

81.10 with JHF66, but same behavoir in other customer with most updated JHF

[Timothy_Hall]

Provide output of free -m please.

[CheckPointerXL]

free -mh
total used free shared buff/cache available
Mem: 30G 22G 580M 49M 7.8G 4.5G
Swap: 31G 1.2G 29G

free -mh from 2-3days ago, just few hours ago cutsomer moved VS to other member, sorry

[Timothy_Hall]

I'd say it is normal if you have a large and busy firewall, yes.  You have 4.5GB of RAM (available) that could be freed at a moment's notice out of your 32GB for code execution.  You have dipped into swap space (1.2G) a bit probably during a policy installation, but when considered against 32GB of RAM it is not too bad.  Might be interesting to run top and hit M to see what processes are using the most memory.  If available memory (not free memory) is constantly dropping over time with no major changes that could indicate a memory leak.

[Matlu]

Hello,

It is normal to see this result in the command "free -m".



I am referring to the value that appears in the "buff/cache" column.

Is there a way to free that value from that column?

I want to make sure, that I don't have problems with the memory of my GW.

Cheers.

[Timothy_Hall]

buff/cache will be freed and reallocated at a moments notice to be "used" by the system if the "free" value drops too low.  There is no way to clear buff/cache other than rebooting, but why would you want to?  Memory that is otherwise sitting around doing absolutely nothing (free) is being temporarily used to cache operations to and from the hard drive (buff/cache), which is several orders of magnitude slower than cached access from RAM.

[Matlu]

Hello,

The "Buff/cache" value, should not cause a "visual alert" of "high memory consumption" in the SmartConsole, right?

This value, that you just explained, is mostly a "reserve" of memory?

Cheers. 🙂

[Timothy_Hall · Accepted solution]

Correct, it is just memory being used to improve system performance.  Some memory reporting/tools commands report the "free" amount as the total available (which is wrong) instead of "available" which is free + buff/cache and much more accurately reflects how much memory is available for the system.