topic Re: Physical memory is high in General Topics

Physical memory is high

Jeff_Gao — Tue, 12 Feb 2019 06:39:36 GMT

Dear all

My CP23500 is 16G memory and traffic is low.

but memory is high,as follow:

This is why?Thanks!

Re: Physical memory is high

Martin_Raska — Tue, 12 Feb 2019 07:04:24 GMT

Hi,

post commands from expert mode:

free -mt

enabled_blades

fw ctl multik stat

Re: Physical memory is high

Jeff_Gao — Tue, 12 Feb 2019 07:52:42 GMT

as follow:

[Expert@FW001:0]# free -mt
total used free shared buffers cached
Mem: 15798 14666 1131 0 394 1577
-/+ buffers/cache: 12695 3103
Swap: 18449 0 18449
Total: 34247 14667 19580

[Expert@FW001:0]# enabled_blades
fw urlf av appi ips anti_bot mon

[Expert@FW001:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 39 | 54 | 6345
1 | Yes | 19 | 62 | 5343
2 | Yes | 38 | 58 | 6763
3 | Yes | 18 | 47 | 5993
4 | Yes | 37 | 61 | 6299
5 | Yes | 17 | 50 | 6063
6 | Yes | 36 | 50 | 5943
7 | Yes | 16 | 61 | 6050
8 | Yes | 35 | 41 | 5736
9 | Yes | 15 | 49 | 6165
10 | Yes | 34 | 43 | 6287
11 | Yes | 14 | 56 | 6063
12 | Yes | 33 | 64 | 5961
13 | Yes | 13 | 57 | 5984
14 | Yes | 32 | 52 | 6067
15 | Yes | 12 | 56 | 5965
16 | Yes | 31 | 55 | 6024
17 | Yes | 11 | 54 | 5940
18 | Yes | 30 | 55 | 5516
19 | Yes | 10 | 45 | 6514
20 | Yes | 29 | 54 | 6390
21 | Yes | 9 | 53 | 6043
22 | Yes | 28 | 45 | 6093
23 | Yes | 8 | 51 | 6032
24 | Yes | 27 | 43 | 5962
25 | Yes | 7 | 46 | 6119
26 | Yes | 26 | 36 | 5992
27 | Yes | 6 | 46 | 5822
28 | Yes | 25 | 55 | 6363
29 | Yes | 5 | 45 | 5815
30 | Yes | 24 | 55 | 6337
31 | Yes | 4 | 58 | 6125
32 | Yes | 23 | 62 | 6111
33 | Yes | 3 | 50 | 6125
34 | Yes | 22 | 58 | 6083
35 | Yes | 2 | 61 | 5984

Re: Physical memory is high

Martin_Raska — Tue, 12 Feb 2019 10:21:55 GMT

I assume there is nothing wrong, you are using a lot of NG blades which consumes memory. The real value of memory consumption without cache is 12695, 1577 is cached and can be freed any time.

Re: Physical memory is high

_Val_ — Tue, 12 Feb 2019 10:37:48 GMT

Second that, there is nothing wrong here. We rely on OS "mem" command to show physical memory utilization. That command with 2.6 kernel it is known to report more than FW really needs as being used. That is normal and should not be causing any alarm unless you start swapping.

Re: Physical memory is high

Timothy_Hall — Tue, 12 Feb 2019 14:51:33 GMT

Agree with Val, as long as second number on the Swap line is 0 everything is running completely within RAM and there is no need for paging/swapping, so your system is running optimally from a memory perspective.

--
"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com

Re: Physical memory is high

Jeff_Gao — Wed, 13 Feb 2019 01:04:31 GMT

I have another firewall,still 23500(R80.10),their functions and policy are the same,and this fw traffic is higher,as follow:

But its memory is very low,as follow:

[Expert@FW002:0]# free -mt
total used free shared buffers cached
Mem: 15798 11985 3812 0 441 3278
-/+ buffers/cache: 8266 7531
Swap: 18449 0 18449
Total: 34247 11985 22261
[Expert@FW002:0]# enabled_blades
fw urlf av appi ips anti_bot mon
[Expert@FW002:0]#
[Expert@FW002:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 39 | 919 | 2617
1 | Yes | 19 | 957 | 2642
2 | Yes | 38 | 953 | 2706
3 | Yes | 18 | 936 | 2553
4 | Yes | 37 | 962 | 2692
5 | Yes | 17 | 917 | 2604
6 | Yes | 36 | 970 | 2635
7 | Yes | 16 | 972 | 2617
8 | Yes | 35 | 978 | 2663
9 | Yes | 15 | 993 | 2596
10 | Yes | 34 | 970 | 2646
11 | Yes | 14 | 980 | 2643
12 | Yes | 33 | 952 | 2640
13 | Yes | 13 | 1018 | 2731
14 | Yes | 32 | 928 | 2624
15 | Yes | 12 | 997 | 2647
16 | Yes | 31 | 962 | 2593
17 | Yes | 11 | 989 | 2609
18 | Yes | 30 | 922 | 2512
19 | Yes | 10 | 884 | 2576
20 | Yes | 29 | 1025 | 2652
21 | Yes | 9 | 1027 | 2754
22 | Yes | 28 | 1071 | 2733
23 | Yes | 8 | 987 | 2633
24 | Yes | 27 | 927 | 2714
25 | Yes | 7 | 1015 | 2651
26 | Yes | 26 | 1017 | 2619
27 | Yes | 6 | 1005 | 2652
28 | Yes | 25 | 1017 | 2651
29 | Yes | 5 | 899 | 2535
30 | Yes | 24 | 1063 | 2791
31 | Yes | 4 | 1043 | 2670
32 | Yes | 23 | 979 | 2751
33 | Yes | 3 | 947 | 2766
34 | Yes | 22 | 988 | 2657
35 | Yes | 2 | 931 | 2679

So,I'm a little confused.

Re: Physical memory is high

Alex- — Wed, 13 Feb 2019 07:48:33 GMT

Do you have the latest hotfix installed?

Re: Physical memory is high

Jeff_Gao — Wed, 13 Feb 2019 07:51:55 GMT

No,hotfix is take_56

Re: Physical memory is high

_Val_ — Wed, 13 Feb 2019 09:50:22 GMT

As said, you can ignore mem output and free memory at all, unless start swapping.

Re: Physical memory is high

Jeff_Gao — Wed, 13 Feb 2019 11:21:45 GMT

I got it,thanks all!

Re: Physical memory is high

good — Thu, 08 Aug 2019 10:14:42 GMT

what is the solution???

Re: Physical memory is high

Maarten_Sjouw — Thu, 08 Aug 2019 21:31:05 GMT

The best solution is to not care about it until you see the Swap memory Used increase above 0.

Re: Physical memory is high

PhoneBoy — Thu, 08 Aug 2019 23:18:55 GMT

As long as you're not swapping, there's nothing to solve.
If you are swapping, post output of some of the commands here and we can have a look.

Re: Physical memory is high

CJ2019 — Wed, 11 Sep 2019 10:06:59 GMT

Seeing similar issue on our primary checkpoint

free -mt
total used free shared buffers cached
Mem: 15849 15199 650 0 241 2336
-/+ buffers/cache: 12621 3228
Swap: 17884 0 17884
Total: 33734 15199 18534

Does it look alright?

Re: Physical memory is high

Timothy_Hall — Sun, 15 Sep 2019 19:00:38 GMT

Yes, the second number on the swap line is zero so your firewall is completely executing in RAM and there is no swapping to disk.

Based on the output, of the available 16GB of RAM your firewall is using 12621 for code execution, 3228 for buffering and caching (this memory can be reallocated for code execution if necessary) and 650 free. So you still have 3878 available for code execution if needed.

Re: Physical memory is high

genisis__ — Wed, 16 Oct 2019 19:22:31 GMT

when issuing the command enabled_blades, clearly this tells us what blades are in use, however is there a way to determine how much memory / resource each blade is actually using?

I have a similar concern regarding memory usage; currently have 64GB of RAM and 40GB is already used. After a reboot this drops to 16GB and then creeps up over two weeks. At this point the concern is low as the swap file is not hit, but I do have two VS's that I'm going to adding the VSX system which have roughly 80,000 concurrent connections each, so have a real concern this is going to put me in a high memory usage scenario.

I suspect a memory leak as I don't see memory being released after core hours.

We are running the following blades:

IPS/AV/ABOT/URLF/FW/Content_Awareness and Monitor.

Re: Physical memory is high

Ryan_St__Germai — Wed, 16 Oct 2019 19:32:54 GMT

We are having the same issue. Looks like a memory leak. We have a smaller Gateway cluster with 16GB of ram. Gateway eventually crashes with errors in /var/log/messages related to memory exhaustion. TAC has is running a memory leak test. I jumped on the following thread where two others have/had a similar issue:
https://community.checkpoint.com/t5/General-Topics/Problem-with-5400-device-after-firmware-upgrade-to-80-30/m-p/64506#M13144

Re: Physical memory is high

Timothy_Hall — Wed, 16 Oct 2019 19:46:09 GMT

Please post output of free -m. The memory creep you are seeing is probably not a memory leak, but free memory being utilized for buffering and caching. This buffering/caching memory use will drop back down after a reboot then slowly grow again which is expected.

Re: Physical memory is high

genisis__ — Thu, 17 Oct 2019 07:16:35 GMT

Hi Timothy,

The hardware is a pair of 15600's with 64GB RAM (MAX), with R80.20 HFA103. Additionally these appliances are running as VSX appliances with 5 active Virtual systems. The following blades are enabled:

fw urlf av appi ips anti_bot content_awareness mon

Top output:

Tasks: 594 total, 1 running, 593 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.0%us, 1.0%sy, 0.0%ni, 94.4%id, 0.1%wa, 0.0%hi, 0.5%si, 0.0%st
Mem: 65747304k total, 40054152k used, 25693152k free, 1965852k buffers
Swap: 33551672k total, 0k used, 33551672k free, 21309224k cached

Clearly above we see no swap being hit.

Uptime:

08:09:03 up 16 days, 11:37, 1 user, load average: 3.44, 3.29, 3.14

But we also see that the uptime is 16 days, initial memory utilisation was around 16GB after reboot.

free -m

             total       used       free     shared    buffers     cached
Mem:         64206      39177      25028          0       1919      20863
-/+ buffers/cache:      16394      47811
Swap:        32765          0      32765

We have at least another 2 virtual systems to implement, assume that each VS will run around 100,000 concurrent connections (200,000 total).

So my concern here is we will not have enough resource to handle this. Additionally I do not see any memory release out side of peak ours when the concurrent connections across all the VS's drop. I would have though that memory should be released.