cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post
Highlighted
abihsot__
Nickel

High memory usage

Hello,

Wanted to share the issue we have with our gateway.  We have following blades enabled:

fw urlf appi identityServer SSL_INSPECT content_awareness mon

Appliance is with 16gb, running latest R80.30.

The problem we are having is that at some point memory usage increases sharply and it never comes down, unless we reboot appliance. This is causing issues to the traffic because some connections are getting disconnected during occurrence. I can't find in top (shift+m) any process which would contribute to this behaviour.

I hope I am not alone with this issue, so please give a shout if you have something similar. Some of the occurrences from the past to show what happens:

 

image.png

image.png

image.png

0 Kudos
11 Replies
Admin
Admin

Re: High memory usage

Note that in general, it is not unusual for an appliance to be utilizing most of its physical memory.
You can also see that a lot of the memory used is actually kernel memory, so you won't necessarily see a process associated with it.

Can you describe in more detail about the connections that disconnect?
What kind of connections are they?
What behaviors do you observe?
What debugging have you done regarding these connections?
0 Kudos
abihsot__
Nickel

Re: High memory usage

Hi,

As I understand disconnected connections are consequence of consumed memory. I couldn't find quickly SK number but it was explaining that GAIA protects itself and cuts some of the connections when such situation arises. Most noticeably some (not all) ssh connections to the servers gets disconnected.

What I observed as well, is when memory hits high consumption, accepted packets and number of connections drops unusually low. This might explain what I found in SK.

So far I did memory leak detection procedure, however this issue occurs once every 2-3 weeks. Memleak procedure says "memory leak plausible", but policy push was done, therefore result might be misleading. TAC wasn't impressed about memleak procedure output as well.

 

 

0 Kudos

Re: High memory usage

Please provide the output of free -m.  As Dameon said it is not unusual for Gaia to allocate free memory for buffering and caching of disk operations on an ongoing basis which accounts for the increasing total utilization. The kernel says it has 8GB free memory in your cpview screenshot...

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
0 Kudos
abihsot__
Nickel

Re: High memory usage

Screenshots I put are not from the very same occurrence. I just wanted to illustrate what is happening. You can see from free -m output that free memory comes to a very little number but cached remains the same.

Mon Sep 9 21:56:31 CEST 2019
total used free shared buffers cached
Mem: 15849 10360 5489 0 190 925
-/+ buffers/cache: 9243 6605
Swap: 17884 6 17878
Mon Sep 9 23:56:45 CEST 2019
total used free shared buffers cached
Mem: 15849 10350 5499 0 203 930
-/+ buffers/cache: 9216 6633
Swap: 17884 6 17878
Tue Sep 10 01:56:59 CEST 2019
total used free shared buffers cached
Mem: 15849 10350 5499 0 215 1017
-/+ buffers/cache: 9116 6733
Swap: 17884 6 17878
Tue Sep 10 03:57:13 CEST 2019
total used free shared buffers cached
Mem: 15849 10291 5557 0 225 1021
-/+ buffers/cache: 9044 6804
Swap: 17884 6 17878
Tue Sep 10 05:57:28 CEST 2019
total used free shared buffers cached
Mem: 15849 10342 5507 0 233 1033
-/+ buffers/cache: 9075 6774
Swap: 17884 6 17878
Tue Sep 10 07:57:42 CEST 2019
total used free shared buffers cached
Mem: 15849 10420 5429 0 240 1080
-/+ buffers/cache: 9098 6751
Swap: 17884 6 17878
Tue Sep 10 09:57:56 CEST 2019
total used free shared buffers cached
Mem: 15849 10663 5186 0 249 1099
-/+ buffers/cache: 9314 6535
Swap: 17884 6 17878
Tue Sep 10 11:58:11 CEST 2019
total used free shared buffers cached
Mem: 15849 15001 847 0 256 1131
-/+ buffers/cache: 13613 2236
Swap: 17884 6 17878
Tue Sep 10 13:58:25 CEST 2019
total used free shared buffers cached
Mem: 15849 15072 776 0 261 1158
-/+ buffers/cache: 13653 2196
Swap: 17884 6 17878

 

This is current situation on a gateway:

total used free shared buffers cached
Mem: 15849 14111 1738 0 423 3349
-/+ buffers/cache: 10337 5511
Swap: 17884 0 17884

As per my understanding we have 3,3GB cached and 1.7GB free which comes to a 5GB available for operating system.

0 Kudos

Re: High memory usage

Correct, looks like you have plenty of memory available to the OS (~5GB) and swap usage is negligible.  When you reboot the buffer/cached values start small and grow as more than more accesses to the disk are performed.  They will eventually top out at around 90% total memory used and not go beyond that point.

 

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
0 Kudos
abihsot__
Nickel

Re: High memory usage

There is something wrong with the gateway and I can't figure out this... We had another occurrence again.

It was working just fine:

image.png

until few moments:

image.png

Please note FW kernel memory is fully used, operating system is using swap and connections/sec dropped to 0.

 

few more screenshots before the incident and after. 

image.png

image.png

Is failed to allocate means it failed because there was no memory available, or it might suggest some hardware problems with memory itself?

Another screenshot might be interesting:

image.png

0 Kudos
Kim_Moberg
Silver

Re: High memory usage

Hi @abihsot__ 

Did you involve TAC?

You must register memory leak detection in the fwkern.conf. This can R&D do for you.

Then you need to keep track on how to reproduce the problem.

I have been using SNMPto keep track on memory states and CPU states which can be recommended.

You can find which OID SNMP tag information here in the sk90860.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

I am using CLI command 'fw ctl pstat' to keep an eye of the memory usages in pct. Keep an eye so percentage isn't above 60%. At 80% usage Check Point services and processes being shutdown.

 

memstat.png

Best Regards
Kim
Tags (2)
0 Kudos
abihsot__
Nickel

Re: High memory usage

Hi Kim,

Yes, TAC was involved but they were useless. I reopened the ticket recently so hopefully will get better engineer this time.

As I mentioned before I did mem leak procedure (parameters in fwkern.conf you are referring), however output of it did not impressed TAC at all, hence no suggestions from them what could cause it...

The issue is so sudden that it might eat the rest of the memory instantly. Did you have memory issues in the past that you are monitoring it closely?

0 Kudos
Kim_Moberg
Silver

Re: High memory usage

@abihsot__ 

I have experienced this issue at some of the EA programs I have been participated in. First let me tell you this have been solved right away and I haven't had any issues afterwards.

We did enabled fwkern parametres for memory leak detection.

This needs to be enabled before you can keep track of which Check Point service or process that is consuming more memory without releasing usage.

I am using these SNMP OID tags to monitor the memory of my gateways.

Challange is that I cannot put it into pct. 

Best Regards
Kim
0 Kudos
Kim_Moberg
Silver

Re: High memory usage

snmp memory.png

Best Regards
Kim
0 Kudos
snowie-swe
Nickel

Re: High memory usage

Hade same issue on appliance box running VSX and HFA5X on R80.30

was not fixed by TAC so moved the VS to another VSX cluster running R80.10.

0 Kudos