High memory utilisation on Threat Emulation box

[salil_arora]

Hi team

Need some guidance with high memory utilisation of one of our TE appliance

Details of the TE appliance:
Platform: TT-10-00
Model: Check Point TE100X
CPU Model: Intel(R) Core(TM) i5-4590S CPU
CPU Frequency: 2993.124 Mhz
Number of Cores: 4
CPU Hyperthreading: Disabled
Number of line cards: 1
Line card 1 model: **bleep**-51040-090
Line card 1 type: 4 ports 1GbE Copper
Number of disks: 1
Disk 1 Model: xxxx
Disk 1 Capacity: 1.00 TB
Total Disks size: 1.00 TB
Total Memory: 16384 MB
Memory Slot 1 Size: 8192 MB
Memory Slot 2 Size: 8192 MB

The box is on HOTFIX_R81_20_JUMBO_HF_MAIN Take: 41 and doesn't have many connections (Peak connections range upto 30)
ID | Active | CPU | Connections | Peak
-----------------------------------------------
0 | Yes | 3 | 5 | 28
1 | Yes | 2 | 7 | 30
2 | Yes | 1 | 5 | 31

Reviewing the top command output with memory sorted I can see lots of memory being consumed by the process gunicorn

Tasks: 326 total, 2 running, 324 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.4 us, 0.4 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.1 hi, 0.0 si, 0.0 st
KiB Mem : 16067604 total, 1906020 free, 10656116 used, 3505468 buff/cache
KiB Swap: 17277900 total, 17231556 free, 46344 used. 4613200 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26689 nobody 20 0 1155900 628800 2744 S 0.0 3.9 1:19.67 python3
27760 admin 20 0 616240 407728 2028 S 0.0 2.5 4:11.65 gunicorn
20443 nobody 20 0 616496 407356 1652 S 0.0 2.5 0:00.13 gunicorn
20444 nobody 20 0 616496 407356 1652 S 0.0 2.5 0:00.13 gunicorn
20445 nobody 20 0 616496 407356 1652 S 0.0 2.5 0:00.13 gunicorn
20446 nobody 20 0 616496 407356 1652 S 0.0 2.5 0:00.13 gunicorn
20447 nobody 20 0 616496 407356 1652 S 0.0 2.5 0:00.13 gunicorn
20918 nobody 20 0 616240 406380 1656 S 0.0 2.5 0:00.22 gunicorn
20914 nobody 20 0 616240 406308 1596 S 0.0 2.5 0:00.21 gunicorn
20915 nobody 20 0 616240 406308 1596 S 0.0 2.5 0:00.22 gunicorn
20916 nobody 20 0 616240 406300 1588 S 0.0 2.5 0:00.22 gunicorn
20917 nobody 20 0 616240 406300 1588 S 0.0 2.5 0:00.22 gunicorn
15012 nobody 20 0 616240 405368 1656 S 0.0 2.5 0:00.31 gunicorn
15009 nobody 20 0 616240 405320 1616 S 0.0 2.5 0:00.31 gunicorn
15010 nobody 20 0 616240 405308 1604 S 0.0 2.5 0:00.31 gunicorn
15013 nobody 20 0 616240 405308 1604 S 0.0 2.5 0:00.31 gunicorn
15011 nobody 20 0 616240 405304 1600 S 0.0 2.5 0:00.31 gunicorn
19764 nobody 20 0 616108 404964 1632 S 0.0 2.5 0:01.22 gunicorn
19737 nobody 20 0 616108 404956 1624 S 0.0 2.5 0:01.20 gunicorn
19740 nobody 20 0 616108 404952 1620 S 0.0 2.5 0:01.21 gunicorn
19753 nobody 20 0 616108 404952 1620 S 0.0 2.5 0:01.22 gunicorn
19757 nobody 20 0 616108 404952 1620 S 0.0 2.5 0:01.21 gunicorn
27266 nobody 20 0 616108 404068 1644 S 0.0 2.5 0:01.30 gunicorn
27267 nobody 20 0 616108 404068 1644 S 0.0 2.5 0:01.31 gunicorn
27268 nobody 20 0 616108 404068 1644 S 0.0 2.5 0:01.30 gunicorn
27269 nobody 20 0 616108 404068 1644 S 0.0 2.5 0:01.32 gunicorn
27270 nobody 20 0 616108 404068 1644 S 0.0 2.5 0:01.30 gunicorn
10685 nobody 20 0 615852 403416 1652 S 0.0 2.5 0:01.56 gunicorn
10687 nobody 20 0 615852 403416 1648 S 0.0 2.5 0:01.57 gunicorn
10688 nobody 20 0 615852 403416 1648 S 0.0 2.5 0:01.59 gunicorn
10684 nobody 20 0 615852 403412 1648 S 0.0 2.5 0:01.57 gunicorn
10686 nobody 20 0 615852 403412 1648 S 0.0 2.5 0:01.57 gunicorn
18659 nobody 20 0 551736 398848 1872 S 0.0 2.5 0:01.68 gunicorn
18660 nobody 20 0 550316 397828 1652 S 0.0 2.5 0:01.66 gunicorn
18661 nobody 20 0 550316 397828 1652 S 0.0 2.5 0:01.65 gunicorn
18662 nobody 20 0 550316 397828 1652 S 0.0 2.5 0:01.66 gunicorn
18663 nobody 20 0 550316 397828 1652 S 0.0 2.5 0:01.67 gunicorn
7697 nobody 20 0 550316 389884 1652 S 0.0 2.4 0:01.72 gunicorn
.........................................................................................................................

The swap memory is being used aswell:

            total    used     free   shared buff/cache available
Mem: 15691  10405  1861    17          3423           4505
Swap: 16872  45       16827

Output of ps -auxw is as below:

PID TTY TIME CMD
14029 pts/2 00:00:00 ps
28687 ? 00:00:00 sshd
28693 pts/2 00:00:00 shell_wrapper.s
28702 pts/2 00:00:00 sudo
28703 pts/2 00:00:00 bash

Output of cpwd_admin list is as below:

APP PID STAT #START START_TIME MON COMMAND
CPVIEWD 13961 E 1 [10:35:09] 3/4/2024 N cpviewd
OTLPAGENT 14021 E 1 [10:35:09] 3/4/2024 N cpview_exporter
CPVIEWS 14028 E 1 [10:35:09] 3/4/2024 N cpview_services
CVIEWAPIS 20712 E 1 [10:44:13] 3/4/2024 N cpview_api_service
SXL_STATD 14036 E 1 [10:35:09] 3/4/2024 N sxl_statd
MSGD 14044 E 1 [10:35:09] 3/4/2024 Y msgd
CPD 14050 E 1 [10:35:09] 3/4/2024 Y cpd
MPDAEMON 14061 E 1 [10:35:09] 3/4/2024 N mpdaemon /opt/CPshrd-R81.20/log/mpdaemon.elg /opt/CPshrd-R81.20/conf/mpdaemon.conf
TP_CONF_SERVICE 14087 E 1 [10:35:09] 3/4/2024 N tp_conf_service --conf=tp_conf.json --log=error
CI_CLEANUP 14314 E 1 [10:35:14] 3/4/2024 N avi_del_tmp_files
CIHS 14316 E 1 [10:35:14] 3/4/2024 N ci_http_server -j -f /opt/CPsuite-R81.20/fw1/conf/cihs.conf
FWD 14325 E 1 [10:35:14] 3/4/2024 N fwd
SPIKE_DETECTIVE 20713 E 1 [10:44:13] 3/4/2024 N spike_detective
RAD 14902 E 1 [10:35:20] 3/4/2024 N rad
DLPU_0 16393 E 1 [10:35:35] 3/4/2024 Y dlpu -i4 0 1 -i6 -1 -1
DLPU_1 16400 E 1 [10:35:35] 3/4/2024 Y dlpu -i4 2 2 -i6 -1 -1
RTMD 26637 E 1 [10:37:31] 3/4/2024 N rtmd
DASERVICE 26832 E 1 [10:37:32] 3/4/2024 N DAService_script
AUTOUPDATER 26855 E 1 [10:37:32] 3/4/2024 N AutoUpdaterService.sh
LPD 14363 E 1 [10:39:32] 3/4/2024 N lpd

enabled_blades are as below:
fw av ThreatEmulation mon Scrub

Output of meminfo:

MemTotal: 16067604 kB
MemFree: 1902576 kB
MemAvailable: 4610896 kB
Buffers: 384640 kB
Cached: 2539212 kB
SwapCached: 4260 kB
Active: 8548532 kB
Inactive: 2595080 kB
Active(anon): 6929856 kB
Inactive(anon): 1307652 kB
Active(file): 1618676 kB
Inactive(file): 1287428 kB
Unevictable: 152 kB
Mlocked: 16 kB
SwapTotal: 17277900 kB
SwapFree: 17231556 kB
Dirty: 144 kB
Writeback: 0 kB
AnonPages: 8215948 kB
Mapped: 122492 kB
Shmem: 17840 kB
Slab: 582576 kB
SReclaimable: 408752 kB
SUnreclaim: 173824 kB
KernelStack: 11696 kB
PageTables: 92324 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 25311700 kB
Committed_AS: 10596024 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 2067964 kB
VmallocChunk: 34357551444 kB
Percpu: 704 kB
AnonHugePages: 1366016 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 557976 kB
DirectMap2M: 16109568 kB
DirectMap1G: 0 kB

Output of cpstat fw -f all:

Product name: Firewall
Major version: 9
Minor version: 9
Kernel build num.: xxx
Policy name:xxxxxxxxxxxx
Policy install time: Fri Apr 12 20:11:38 2024
Num. connections: 11
Peak num. connections: 533
Connections capacity limit: 0
Total accepted packets: 91134456

Interface table
-------------------------------------------
|Name |Dir|Accept |Drop|Reject|Log |
-------------------------------------------
|Mgmt |in | 5218781|2900| 0| 7356|
|Mgmt |out|83840391| 0| 0| 1334|
|bond1 |in | 0| 0| 0| 0|
|bond1 |out| 0| 0| 0| 0|
|bond1.xxxx|in | 1159509| 459| 0| 284|
|bond1.xxxx|out| 915775| 44| 0| 8737|
-------------------------------------------
| | |91134456|3403| 0|17711|
-------------------------------------------

Interface table (64-bit)
-------------------------------------------
|Name |Dir|Accept |Drop|Reject|Log |
-------------------------------------------
|Mgmt |in | 5218781|2900| 0| 7356|
|Mgmt |out|83840391| 0| 0| 1334|
|bond1 |in | 0| 0| 0| 0|
|bond1 |out| 0| 0| 0| 0|
|bond1.xx|in | 1159509| 459| 0| 284|
|bond1.xxx|out| 915775| 44| 0| 8737|
-------------------------------------------
| | |91134456|3403| 0|17711|
-------------------------------------------

ISP link table
------------------
|Name|Status|Role|
------------------
------------------

hmem - block size: 4096
hmem - requested bytes: 1233125376
hmem - initial allocated bytes: 1233125376
hmem - initial allocated blocks: 0
hmem - initial allocated pools: 0
hmem - current allocated bytes: 1233125376
hmem - current allocated blocks: 301056
hmem - current allocated pools: 1
hmem - maximum bytes: 1281359872
hmem - maximum pools: 512
hmem - bytes used: 0
hmem - blocks used: 0
hmem - bytes unused: 1233125376
hmem - blocks unused: 301056
hmem - bytes peak: 218223496
hmem - blocks peak: 55811
hmem - bytes internal use: 0
hmem - number of items: 0
hmem - alloc operations: 245917421
hmem - free operations: 243930260
hmem - failed alloc: 0
hmem - failed free: 0
kmem - system physical mem: 0
kmem - available physical mem: 0
kmem - aix heap size: 0
kmem - bytes used: 1695331560
kmem - blocking bytes used: 9851712
kmem - non blocking bytes used: 1685479848
kmem - bytes unused: 0
kmem - bytes peak: 1741700516
kmem - blocking bytes peak: 9941124
kmem - non blocking bytes peak: 1731759392
kmem - bytes internal use: 108112
kmem - number of items: 6757
kmem - alloc operations: 1675154
kmem - free operations: 1668397
kmem - failed alloc: 0
kmem - failed free: 0
inspect - packets: 0
inspect - operations: 0
inspect - lookups: 0
inspect - record: 0
inspect - extract: 0
cookies - total: 371979711
cookies - alloc: 0
cookies - free: 0
cookies - dup: 332666019
cookies - get: 1129273158
cookies - put: 363541903
cookies - len: 743931873
chains - alloc: 0
chains - free: 0
fragments - fragments: 2
fragments - expired: 0
fragments - packets: 1
ufp - % hits ratio: 0
ufp - total connections: 0
ufp - hits connections: 0
ufp - session max: 0
ufp - session current: 0
ufp - session count: 0
ufp - rej session : 0
ufp - time stamp:
ufp - is alive: 0
http - pid: 0
http - proto: 0
http - port: 0
http - logical port: 0
http - max avail socket: 0
http - socket in use max: 0
http - socket in use current: 0
http - socket in use count: 0
http - session max: 0
http - session current: 0
http - session count: 0
http - auth session max: 0
http - auth session current: 0
http - auth session count: 0
http - accepted session: 0
http - rejected session: 0
http - auth failures: 0
http - opsec cvp session max: 0
http - opsec cvp session current: 0
http - opsec cvp session count: 0
http - opsec cvp rej session : 0
http - ssl encryp session max: 0
http - ssl encryp session current: 0
http - ssl encryp session count: 0
http - transparent session max: 0
http - transparent session current: 0
http - transparent session count: 0
http - proxied session max: 0
http - proxied session current: 0
http - proxied session count: 0
http - tunneled session max: 0
http - tunneled session current: 0
http - tunneled session count: 0
http - ftp session max: 0
http - ftp session current: 0
http - ftp session count: 0
http - time stamp:
http - is alive: 0
ftp - pid: 0
ftp - proto: 0
ftp - port: 0
ftp - logical port: 0
ftp - max avail socket: 0
ftp - socket in use max: 0
ftp - socket in use current: 0
ftp - socket in use count: 0
ftp - session max: 0
ftp - session current: 0
ftp - session count: 0
ftp - auth session max: 0
ftp - auth session current: 0
ftp - auth session count: 0
ftp - accepted session: 0
ftp - rejected session: 0
ftp - auth failures: 0
ftp - opsec cvp session max: 0
ftp - opsec cvp session current: 0
ftp - opsec cvp session count: 0
ftp - opsec cvp rej session : 0
ftp - time stamp:
ftp - is alive: 0
telnet - pid: 0
telnet - proto: 0
telnet - port: 0
telnet - logical port: 0
telnet - max avail socket: 0
telnet - socket in use max: 0
telnet - socket in use current: 0
telnet - socket in use count: 0
telnet - session max: 0
telnet - session current: 0
telnet - session count: 0
telnet - auth session max: 0
telnet - auth session current: 0
telnet - auth session count: 0
telnet - accepted session: 0
telnet - rejected session: 0
telnet - auth failures: 0
telnet - time stamp:
telnet - is alive: 0
rlogin - pid: 0
rlogin - proto: 0
rlogin - port: 0
rlogin - logical port: 0
rlogin - max avail socket: 0
rlogin - socket in use max: 0
rlogin - socket in use current: 0
rlogin - socket in use count: 0
rlogin - session max: 0
rlogin - session current: 0
rlogin - session count: 0
rlogin - auth session max: 0
rlogin - auth session current: 0
rlogin - auth session count: 0
rlogin - accepted session: 0
rlogin - rejected session: 0
rlogin - auth failures: 0
rlogin - time stamp:
rlogin - is alive: 0
smtp - pid: 0
smtp - proto: 0
smtp - port: 0
smtp - logical port: 0
smtp - max avail socket: 0
smtp - socket in use max: 0
smtp - socket in use current: 0
smtp - socket in use count: 0
smtp - session max: 0
smtp - session current: 0
smtp - session count: 0
smtp - accepted session: 0
smtp - rejected session: 0
smtp - mail max: 0
smtp - mail curr: 0
smtp - mail count: 0
smtp - outgoing mail max: 0
smtp - outgoing mail curr: 0
smtp - outgoing mail count: 0
smtp - max mail on conn: 0
smtp - total mails : 0
smtp - time stamp: Sat Apr 13 08:11:40 2024
smtp - is alive: 0
sync - configured: No
sync - out state: Off
sync - in state: Off
sync - number of sent packets: 0
sync - number of Kbytes sent: 0
sync - number of packets received: 0
sync - number of Kbytes received: 0
sync - number of retrans requests sent: 0
sync - number of retrans requests received: 0
sync - number of ack packets sent: 0
sync - number of ack packets received: 0
sync - number of packets dropped by network: 0
sync - overall number of table updates to be synced: 152488
sync - number of updates filtered by 'non sync': 0

I have restarted the box once which helped to resolve the memory utilisation however it has now gone back up to 80% and its constantly increasing

Anything you recommend for me to try to resolve this high memory utilisation please?

[PhoneBoy]

Are there any actual symptoms of high memory usage (like performance/functionality degradation)?
If you think you have a memory leak, use this SK to diagnose where and engage with TAC: https://support.checkpoint.com/results/sk/sk35496

However, what you're describing is normal behavior.
"free" only shows you what has been allocated thus far, which will normally decrease and not increase.
This is normal.
"buff/cache" shows you what memory is allocated for other things but could easily be freed up if necessary.
This will fluctuate. 

If both of these things are near their maximum, then swap will be utilized.
For gateways passing production traffic, swap should not be used or you'll likely start experiencing performance-related issues.
Threat Emulation appliances aren't inline, so swap is less problematic here, but the usage is minimal.

[salil_arora]

Hi mate
Thanks for your reply
There are currently no symptoms yet as I restarted the TE appliance when the memory went as high as 86% and since the restart the memory started going up from 50% all the way upto 80% now and it is still increasing. With this trend the memory will keep increasing hence I wanted to do something about it before there is a real impact
I will have a look at the memory leak article and see if I can spot something
Thank you

[Lesley]

I don't see an issue. Unless you have memory leak as stated by PhoneBoy. If there is no memory leak then it is OK.

Linux systems uses all the memory it can or most of it. The trick is to monitor swap. But in this case it is to low to even consider to be worried. 

[salil_arora]

Hi mate
There is not an issue now however there can be as the memory keeps increasing day by day.
I will have a look at the memory leak article
Thanks for your input

[Timothy_Hall]

You are fine, ignore what "free" is reporting (1.8GB) and focus on what "available" is reporting (4.5GB), which is a more accurate representation of how much memory is really available.  Transient conditions such as policy installations can cause high memory utilization which may cause the system to temporarily dip into swap space.  However it never frees that swap space until reboot.  Please see my Be your Own TAC speech from CPX Vegas where I discussed how to properly assess the amount of memory available on a system.

[salil_arora]

Thank you so much Timothy for your update,the available memory has gone down to 4GB now
Will look into your CPX speech now and see if I can diagnose further

[Chris_Atkinson]

Recommend upgrading to JHF T53 and monitoring.

There were some memory leaks addressed between these takes that may apply depending on exactly how the TE appliance is deployed.

[salil_arora]

Thanks Chris for your input and suggestion. The funny thing is we have multiple TE 's with same Take(T43) and only one TE appliance out of the 4 is facing the memory issue. Looking into this further I see this also:

[Expert@xxxxxxxxxxx:0]# ps -ef |grep "ifi"

nobody 3481 27708 0 Apr10 ? 00:00:43 /var/log/py/python3_64 /var/log/py/rpi/gunicorn/gunicorn --config /var/log/py/rpmi/server_conf/ifi_server_config.py
admin 8104 13212 0 14:08 pts/2 00:00:00 grep --color=auto ifi
admin 27708 25467 0 Apr03 ? 00:00:53 /var/log/py/python3_64 /var/log/py/rpi/gunicorn/gunicorn --config /var/log/py/rpmi/server_conf/ifi_server_config.py
admin 28343 1 0 Apr03 ? 00:00:00 /tmp/ifiPython3 /tmp/ifi_server restart
admin 28347 28343 0 Apr03 ? 00:00:55 /tmp/ifiPython3 /tmp/ifi_server restart
admin 28348 28343 0 Apr03 ? 00:00:56 /tmp/ifiPython3 /tmp/ifi_server restart
admin 28349 28343 0 Apr03 ? 00:00:53 /tmp/ifiPython3 /tmp/ifi_server restart

Reading the release note of Take 43 I see its a known issue:

PRJ-46118, PMTR-91889

Threat Emulation

Multiple ifiPython3 processes may utilize the Security Gateway memory, affecting the Anti-Virus blade performance.

Can you please confirm if the above has been fixed with T53?

Appreciate your help in advance

[Chris_Atkinson]

Are all the TE appliances the same model or different...

Confirmed. Whilst there are rare exceptions, T53 will include fixes of the previous takes as a rule.

[salil_arora]

Thanks for the confirmation Chris,appreciate it
Yes correct we have a mixture of 1000X and 2000X TE
Will go ahead and upgrade the TE to Take 53 as per your recommendation

[Chris_Atkinson]

The different models would have significantly more memory available and potentially a greater ability to endure any leak for longer as a result - assuming one exists.

Work with TAC if you need to put some science to it, there is a procedure (SK) to trace potential leaks otherwise

[salil_arora]

Yeah that makes sense thanks Chris
Yes will do if after upgrading the TE to Take 53 if I face the same symptom(hopefully not)
I will then follow the steps on sk35496
Thanks again for your help