Why is memory swapping on the gateway ?

[Christophe_SIEB]

Hello,
See this post   for the beginning of this thread and the relevant screenshots.
I'm posting here to have clues to understand why memory is swapping on our active gateway.

We have a cluster of two 5400 appliances and one SMS server to handle the management, logging and monitoring.
These gateways are new and running for 2 weeks now (we were not on Check Point before) and we observe recently some swapping on the active gateway, after days of continuous growth of memory consumption (about 2% by day). After 80%, The SMS showed memory consumption in red color. Running top command on the active gateway and sorting on memory (hitting M) shows that the wstlsd process is consuming (or reserving ?) a fair amount of memory. And this amount consumed seems to stay at the same level at night or during non-business days.

Running a "show version all" on the active gateway gives:
Product version Check Point Gaia R80.10
OS Build 462
OS kernel version 2.6.18-92cpx86_64
OS edition 64-bit

We have both HTTPS inspection AND "Categorize HTTPS sites" enabled in AC & URL filtering (I understood that the latter is not working if the former is enabled). The vast majority of https traffic is by now bypassed and we are in the process of gradually deploying root certicate on computers to actively inspect the ssl connections.

To my knowledge, the result of tha command "sar -W" (screenshot taken today) tend to show that the system is actively swapping .

Is the wstlsd process allocating memory even if it does not use it ? Is it possible to have a list of the connections handled by this process ?

Thanks

[PhoneBoy]

Categorize HTTPS Sites and HTTPS Inspection are mutually exclusive options, FYI.

wstlsd handles connections that are being HTTPS Inspected.

You can debug it using the following SK if needed: How to debug WSTLSD daemon 

[Timothy_Hall]

It is normal to see a low amount of pswpins/s with sar -W as processes read various types of data off the hard drive such as code pages, libraries, conf files, etc.  Default pagesize in Linux is 4KB, and the highest number shown in your output is 0.53 pages swapped in per second, or approximately 2KB per second being read from the hard drive which is inconsequential.

Notice that pswpout/s is always zero in your screenshot which means no pages of memory are getting involuntarily thrown out of memory to the hard drive due to a shortage of RAM.  When this number is nonzero then you have some cause for concern.

The memory growth you are seeing is probably due to use of RAM for buffering/caching of hard drive operations, please post output of free -m. 

As Dameon Welch Abernathy‌ noted, wstlsd is related to HTTPS Inspection and the "Categorize HTTPS Sites" checkbox and was covered in my book.  While the memory allocated by each wstlsd process may look concerning, a large amount of that memory is actually being shared amongst the wstlsd processes since they are all doing pretty much the same operations.  So the overall amount of memory being consumed by these processes can appear far higher than it is in actuality.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Christophe_SIEB · Accepted solution]

Hello, the gateway is no longer swapping since last JHA n°103 was applied (all rules and parameters unchanged). See picture. Thanks.