Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Bronze

High memory usage

Hello,

Wanted to share the issue we have with our gateway.  We have following blades enabled:

fw urlf appi identityServer SSL_INSPECT content_awareness mon

Appliance is with 16gb, running latest R80.30.

The problem we are having is that at some point memory usage increases sharply and it never comes down, unless we reboot appliance. This is causing issues to the traffic because some connections are getting disconnected during occurrence. I can't find in top (shift+m) any process which would contribute to this behaviour.

I hope I am not alone with this issue, so please give a shout if you have something similar. Some of the occurrences from the past to show what happens:

 

image.png

image.png

image.png

0 Kudos
42 Replies
Highlighted

Sorry but there is no such command neither in bin/bash nor in /etc/cli.sh 🙂
0 Kudos
Highlighted

What about 'cat /proc/meminfo' ?

0 Kudos
Highlighted

memin.JPG

Here it is? Any ideas?

0 Kudos
Highlighted

Hmm, i believed all R80.40 would use the Dynamic Clish(sk144112) already, my mistake.
In this case the command is "fw ctl pstat".

0 Kudos
Highlighted

Notice that almost all available memory is in Active state which means it is constantly being pulled in for use as cache or buffers. It can and it will be used by the kernel if it runs short on process memory but only if very necessary.

How is the system performance otherwise? What 'top | head -n 10' shows ?

0 Kudos
Highlighted

top_java.JPG

As I mentioned, this java keep using memory.

0 Kudos
Highlighted

Hi,

We have a similar issue but not sure if others have the same outputs.

Running R80.30 Take 155.

On top command I see high usage of memory as well as 'free -m' output;

top - 08:52:30 up 26 days, 21:03, 5 users, load average: 1.79, 2.20, 2.19
Tasks: 342 total, 3 running, 339 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 0.6%sy, 0.3%ni, 91.1%id, 0.0%wa, 0.1%hi, 7.3%si, 0.0%st
Mem: 32784868k total, 31227084k used, 1557784k free, 578016k buffers
Swap: 33551744k total, 196k used, 33551548k free, 20191296k cached

 

# free -m
total used free shared buffers cached
Mem: 32016 30494 1521 0 564 19718
-/+ buffers/cache: 10212 21804
Swap: 32765 0 32765

 

On the other hand when I check from the cpview I see only %17 of usage , also Smart Console shows low memory usage.

1.JPG

2.JPG

 

# fw ctl pstat

System Capacity Summary:
Memory used: 17% (4098 MB out of 24012 MB) - below watermark
Concurrent Connections: 11179 (Unlimited)
Aggressive Aging is enabled, not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 4279132160 bytes in 1044710 (4096 bytes) blocks using 2 pools
Initial memory allocated: 2516582400 bytes (Hash memory extended by 1762549760 bytes)
Memory allocation limit: 20142096384 bytes using 512 pools
Total memory bytes used: 0 unused: 4279132160 (100.00%) peak: 3975273208
Total memory blocks used: 0 unused: 1044710 (100%) peak: 1024447
Allocations: 405809094 alloc, 0 failed alloc, 390362140 free

System kernel memory (smem) statistics:
Total memory bytes used: 5775472876 peak: 6295815528
Total memory bytes wasted: 44266902
Blocking memory bytes used: 34193880 peak: 39109936
Non-Blocking memory bytes used: 5741278996 peak: 6256705592
Allocations: 146815898 alloc, 0 failed alloc, 146791213 free, 0 failed free
vmalloc bytes used: 5706239208 expensive: no

Kernel memory (kmem) statistics:
Total memory bytes used: 2973191124 peak: 5738297476
Allocations: 552592884 alloc, 0 failed alloc
537127921 free, 0 failed free
External Allocations: 1991784 for packets, 30309726 for SXL

0 Kudos
Bronze

Hi,

its fine, don't worry. Most of your memory is cached and is ready to be used if needed. If you really have memory issues you should see high peak values in Advanced->memory->overview. Also when memory is fully consumed you should start getting failure to allocate messages in advanced->memory->smem-failures.

Check my screenshots in previous posts.

By the way, I heard that checkpoint has fixed some bad memory leaks in R80.30 JHF16x. We are running JHF155 and looking forward to JHF191, so hopefully we won't have those nasty issues anymore...

Highlighted

Thanks for the Reply,

 

I did check advanced->memory->smem-failures. but didn't see anything while there were complaints yesterday.Still don't see anything on that screen.

We have 2 clusters running r80.30 T155 and both are the same and was planning to do a new upgrade but bit curious to do it after this even though it may look normal.

Why is the cpview different , does cpview exclude the cashed memory as also snmp outputs on our monitoring tool is getting the data from the cpview as it doesn't alert us.

0 Kudos
Highlighted
Bronze

If there were complains, why would you think it was memory related?

You are running 32GB system and kmem peak was ~5GB. You had plenty of memory.

Memory is a complex thing having many parameters. You should compare apples to apples, so double-check what you were monitoring in your tool.
Highlighted

 

People were complaining of connection issues which I believe it will be user connection problem but was curious when digging in.

One last thing, on the top command there is a swap output which looks it is being used different than the 'free -m' output. Is this normal?because I don't see this usage on the firewalls running R80.10.

3.JPG

 

0 Kudos
Highlighted
Bronze

Yes, I have it too on some gateways. Difference comes from here.

# cat /proc/meminfo |grep -i swap
SwapCached: 0 kB
SwapTotal: 18836204 kB
SwapFree: 18836072 kB

Mem: 16230576k total, 14246328k used, 1984248k free, 333276k buffers
Swap: 18836204k total, 132k used, 18836072k free, 5901604k cached

Maybe there is a way to find out which process allocated that swap space, but really you shouldn't worry about 200Kb.

Highlighted

You're correct , thanks for the replies.
I prefer checking the memory usage from the cpview values as our monitoring and also the Smart console dashboard is also showing these values.
0 Kudos