Memory cache usage on 16200 appliance – expected behavior?

[jcdias]

Hi everyone,

I’m running a Check Point 16200 appliance and noticed that the memory cache has grown from around 31GB to 74GB over the past 3 months. The system has 128GB of RAM.

Most of the memory usage comes from fwk processes, which I assume is normal given the active blades.

Is this cache growth considered normal behavior in Gaia OS over time?

I’m attaching a memory monitoring graph for reference.

Thanks in advance!

PS AUX

ps aux -sort=%mem | head -n 10
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
admin     94261  101  2.2 4069944 2997472 ?     S<Ll May10 189348:59 fwk
admin     94260  7.4  2.0 3824720 2729596 ?     S<Ll May10 13986:45 fwk
admin     94258  2.6  1.9 3650852 2553348 ?     S<Ll May10 4999:42 fwk
admin     94259 46.8  1.0 2476156 1381728 ?     S<Ll May10 87775:36 fwk
admin     94264  2.1  0.9 2305432 1204688 ?     S<Ll May10 4104:42 fwk
admin     94265  6.3  0.9 2277500 1191192 ?     S<Ll May10 11972:38 fwk
admin     94262 10.7  0.9 2267852 1180196 ?     S<Ll May10 20073:20 fwk
admin     68053  0.0  0.3 447812 421192 ?       SLs  May10   0:00 fwk_forker
admin     64700  0.2  0.3 413800 409380 ?       Ss   May10 474:10 /bin/monitord

Addtional info about the appliance:

>cpinfo -y all

This is Check Point CPinfo Build 914000250 for GAIA

[MGMT]
HOTFIX_R81_20_JUMBO_HF_MAIN Take: 92
[IDA]
No hotfixes..
[CPFC]
HOTFIX_TEX_ENGINE_R8120_AUTOUPDATE
[FW1]
HOTFIX_INEXT_NANO_EGG_AUTOUPDATE
HOTFIX_TEX_ENGINE_R8120_AUTOUPDATE
HOTFIX_R81_20_JUMBO_HF_MAIN Take: 92
HOTFIX_GOT_TPCONF_AUTOUPDATE
HOTFIX_R80_40_MAAS_TUNNEL_AUTOUPDATE
HOTFIX_PUBLIC_CLOUD_CA_BUNDLE_AUTOUPDATE

FW1 build number:
This is Check Point's software version R81.20 - Build 043
kernel: R81.20 - Build 050

[Chris_Atkinson]

I'd say normal yes and somewhat dependent on how busy the gateway / environment.

Per sk32206 the Cache becomes available for reuse if the system needs to free memory so typically doesn't indicate a problem.

How much traffic does the gateway handle and which blades are active?

For what it's worth there are fixes to various memory conditions in newer JHF takes higher than T92.

[the_rock]

I know customer that runs one of those on R81.20 jumbo 113, have not heard any issues. They also have 128 GB of ram.

Andy

[G_W_Albrecht]

Recommended is Jumbo Take 113

[_Val_]

Can you share the output of "fw ctl pstat"

[jcdias]

fw ctl pstat

Virtual System Capacity Summary:
Physical memory used: 13% (14377 MB out of 108837 MB) - below watermark
Kernel memory used: 2% (3072 MB out of 108837 MB) - below watermark
Virtual memory used: 0% (312 MB out of 108837 MB) - below watermark
Used: 312 MB by FW, 1152 MB by zeco
Concurrent Connections: 0% (80 out of 14900) - below watermark
Aggressive Aging is enabled, not active

Kernel memory (kmem) statistics:
Total memory bytes used: 90688610 peak: 92443450
Allocations: 1552939106 alloc, 0 failed alloc
1298515489 free, 0 failed free

Cookies:
767363283 total, 0 alloc, 0 free,
0 dup, 393489414 get, 1442259318 put,
1305907202 len, 4713314 cached len, 0 chain alloc,
0 chain free

Connections:
5220704 total, 2707474 TCP, 1780518 UDP, 732696 ICMP,
16 other, 0 anticipated, 4 recovered, 80 concurrent,
430 peak concurrent

Fragments:
258852 fragments, 129426 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
1171909070/0 forw, 267851060/0 bckw, 1435676828 tcpudp,
4083267 icmp, 869311-868742 alloc

Sync: Run "cphaprob syncstat" for cluster sync statistics.

[_Val_ · Accepted solution]

This is your answer:

Virtual System Capacity Summary:
Physical memory used: 13% (14377 MB out of 108837 MB) - below watermark
Kernel memory used: 2% (3072 MB out of 108837 MB) - below watermark
Virtual memory used: 0% (312 MB out of 108837 MB) - below watermark
Used: 312 MB by FW, 1152 MB by zeco
Concurrent Connections: 0% (80 out of 14900) - below watermark
Aggressive Aging is enabled, not active

Kernel memory (kmem) statistics:
Total memory bytes used: 90688610 peak: 92443450
Allocations: 1552939106 alloc, 0 failed alloc
1298515489 free, 0 failed free

You are doing just fine on RAM, no swaps, no high watermarks

[the_rock]

If I were you, I would set connections limit to automatic in smart console fw object.

Andy

[Chris_Atkinson]

Suspect this is a VSX gateway where this isn't possible.

[the_rock]

I always forget the fact you cant do that on vsg gateway...my bad Chris. 

Andy

[the_rock]

Also, can you share what it shows for below commands?

free -g

cpview (look for memory stats)

top

Andy

[Bob_Zimmerman]

Specifically, in 'free -g' (I personally prefer 'free -h', but it does take more thought to interpret), the most important numbers are the Mem row's available, and the Swap row's used.

Free RAM is wasted RAM, so Linux caches stuff. A healthy Linux system should have around 100-200 MB of free RAM, much more available RAM, and no swap being used.

[_Val_]

Exactly that, OS lists lots of RAM not free, but also does not use it much

[jcdias]

Thanks for the help guys