Healthcheck script results

[Vincent_Bacher]

Hello mates,

i just used the healthcheck script on a productive system to see if it's useful and just got the first unsightly result:

+-----------------------+-------------------------------+---------------+
| Memory                | Physical Memory               | WARNING       |
|                       | Swap Memory                   | WARNING       |
+-----------------------+-------------------------------+---------------+

Memory Info:
MemTotal:      7942612 kB
RawMemTotal:         0 kB
MemFree:        150880 kB
Buffers:        205464 kB
Cached:        1345828 kB
SwapCached:     311112 kB
Active:        6972028 kB
Inactive:       569056 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      7942612 kB
LowFree:        150880 kB
SwapTotal:    17824108 kB
SwapFree:     15392428 kB
Dirty:           22896 kB
Writeback:           0 kB
AnonPages:     5986600 kB
Mapped:         134840 kB
Slab:           186292 kB
PageTables:      26524 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  21795412 kB
Committed_AS: 10653012 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    272044 kB
VmallocChunk: 34359466103 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

this is strange because there is just one gateway Cluster managed by this sms.

This is a

Platform: ST-10-00
Model: Smart-1 210

running R80.10 jumbo take 40

Yes, it's not realy most recent take but actually we don't have any issues. Or are there any fixes included in newer takes leading to better memory usage?

Any ideas?

Another interesting Point is this

+-----------------------+-------------------------------+---------------+
| Processes             | Zombie Processes              | OK            |
|                       | Process Restarts              | WARNING       |
+-----------------------+-------------------------------+---------------+

cpwd_admin list
APP        PID    STAT  #START  START_TIME             MON  COMMAND
CPVIEWD    2731   E     1       [13:04:27] 13/9/2017   N    cpviewd
CPD        2755   E     1       [13:04:27] 13/9/2017   Y    cpd
FWD        2878   E     1       [13:04:35] 13/9/2017   N    fwd -n
FWM        2881   E     1       [13:04:36] 13/9/2017   N    fwm
SOLR       11314  E     1       [12:11:15] 3/1/2018    N    java_solr /opt/CPrt-R80/conf/jetty.xml
RFL        3180   E     1       [13:04:40] 13/9/2017   N    LogCore
SMARTVIEW  3198   E     1       [13:04:40] 13/9/2017   N    SmartView
INDEXER    3676   E     1       [14:55:29] 27/11/2017  N    /opt/CPrt-R80/log_indexer/log_indexer
CPM        3331   E     1       [13:04:41] 13/9/2017   N    /opt/CPsuite-R80/fw1/scripts/cpm.sh -s
SMARTLOG_SERVER 3728   E     1       [14:55:29] 27/11/2017  N    /opt/CPSmartLog-R80/smartlog_server
DASERVICE  17592  E     1       [16:48:48] 8/2/2018    N    DAService_script
CPSM       7755   E     1       [13:08:35] 13/9/2017   N    cpstat_monitor
LPD        12544  E     1       [01:22:54] 12/2/2018   N    lpd

Not sure why this led to a warning

Cheers
Vincent

[Christophe_SIEB]

Hello, see my post https://community.checkpoint.com/message/14189-memory-status-shows-red-color-on-management-server , seems to be the same situation on R80.10 Take_70. CPU is low. Memory is 88% today, no swap at the moment, no problems for end users, but i will check if the gateway will swap in the next few days. Regards.

[Vincent_Bacher]

Hello,
thank you for that hint. It's just irritating why in the dasboard it's shown as "green" and the script shows "warning".
Usage is alwas round about 79%
And as there is just one cluster is managed by this smartcenter, this Memory usage is bit high i think
Cheers
Vincent

[Timothy_Hall]

Some tools do not differentiate between memory used for actual execution vs. buffering/caching.  Memory allocated for buffering/caching helps the system run more efficiently, and can be instantly reallocated for execution if needed.  The memory statistics you posted show about 6GB of the 8GB of RAM used for execution with the remainder 2GB being reported as free or being utilized for buffering/caching. 

On an R80+ SMS in particular, I've noticed that java tends to soak up all the memory it can to optimize its own operation, but never quite enough to impede the overall system.  The typical RAM utilization for actual SMS code execution seems to hover around 70-80%, and I'm not sure if java is sensing the amount of memory in the system and taking what it think is its fair share, or there is some other limiting factor in play (such as the -Xmx option to limit the maximum heap size).

This behavior seems quite acceptable on an SMS and shouldn't cause any problems as long as the system is not frequently dipping into swap space, easiest way to determine this is with free -m.  Running top and hitting capital M can also show you the top consumers of memory on an SMS which will be java processes.  Java is not used on the gateway end of things so a shortage of memory there is quite different from this observed behavior on the SMS.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Vincent_Bacher]

Hi Tim,

thanks a lot for that explanation.

Java is a bothersome as i am always wondering if it just needs so much memory or if it does not corectly release unused memory.

In this case it really uses swap space which isn't very nice

# free -m
             total       used       free     shared    buffers     cached
Mem:          7756       7641        114          0        268       1600
-/+ buffers/cache:       5772       1983
Swap:        17406       3107      14298

best
Vincent

[Timothy_Hall]

The reported use of swap space (3107 in your case) seems to indicate that at some point the SMS's memory utilization ballooned ~3GB beyond the amount of physical RAM available.  However even after the actual memory utilization goes back down and everything is efficiently running inside RAM again, that reported swap number never seems to go back down unless the system is rebooted. 

Given that the third line of your free output is reporting 1983 free, I highly doubt the system is actively dipping into swap space.  To confirm this run sar -W to look at historical swap statistics for today (I'm guessing you will see all 0's which is good and means everything is fitting into RAM), or run sar -W 5 5 to see real-time swap statistics.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Vincent_Bacher]

Thanks Tim. Will check that next week.

[Diego_Lopez]

Anyone else experiencing this issue -- Any time I run the healthcheck on any of my R80.10 Gateways/Manager I always seem to get that WARNING on the process restart time checks, almost always it points to this -- DAService_script

Any ideas?

Thank you.

DL

[Timothy_Hall]

The Deployment Agent (a.k.a. CPUSE) automatically updates itself with the latest build and obviously must restart itself every time that happens.  That is probably what you are seeing. Anytime a new firewall software release comes out the Deployment Agent seems to get updated a little more often than usual as the latest fixes are added to the DA to help ensure successful upgrades to the new release.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Diego_Lopez]

Anytime I run the healthcheck scrip on my Primary Smart Center running R80.20 on VMWare Hypervisor, I keep seeing WARNING for Zombie processes -- cpm.sh

+-----------------------+

| Processes |

+-----------------------+

1 zombie processes found.

PID COMMAND

17393 [cpm.sh] <defunct>       

Thank you folks,

-DL