What amount of swap file usage is reasonable / expected?

[ascoyne]

We run two 6000 appliances in HA that are running R81.20.

We monitor with SolarWinds and see the swap file usage on the Active gateway gradually creep up.  Eventually, we have to failover to the Standby gateway as we start to get remote VPN users unable to connect and performance issues.

What should the expected use of a swap file be?

[Hugo_vd_Kooij]

My simple Rule of thumb is what we use for monitoring at the moment:

Any swap usage 0ver 1 GB is cause for concern (WARNING). Any swap usage over 2 GB is cause for grave concern (CRITICAL).

[Bob_Zimmerman]

I would go a little further. To me, any swapping is cause for a warning. RAM is cheap. Just add more.

I don't know what processor the 6700 uses. The 6600 definitely uses an i5-9500E, and the 6800 definitely uses a Xeon E5-2640 v4. If it's consumer-grade (i5 or i7), it's probably officially limited to 32 GB and non-ECC. It remains a source of amusement to me that the 3100 and 3200 support up to 64 GB with ECC, and the 3600 and 3800 support up to 256 GB with ECC, but most of the much-more-expensive 5000 and 6000 series boxes have much lower RAM limits.

[Oliver_Fink]

Just from a very bad experience: On VSX swapping is deadly. Got that problem when migrating from 80.30 to 81.10, where memory consumption was higher than before: same configuration, more memory needed. Suddenly, memory was too small. Not to mention the memory leak we had in the beginning.

Consequence: The VSX cluster became quite unresponsive, sessions were interrupted. 😕

Need to mention: The 15600s still have rotating rust instead of SSD.

[Chris_Atkinson]

Which model of 6000 appliance do you have and what is the current RAM population?

[ascoyne]

We have 6700 appliances.  16GB RAM.  Running R81.20 JHF Take 26.  

[Chris_Atkinson]

This appliance model can have up to 32GB which may be appropriate depending on your combination of traffic volume / enabled blades.

It may also be helpful to check the Jumbo documentation in case there are relevant memory optimization or memory leak fixes in more recent takes etc

[Timothy_Hall · Accepted solution]

I'd agree with @Bob_Zimmerman that ideally swap space used on a gateway should be zero, which means the gateway has no need for paging and swapping at all which maximizes efficiency.  However there can be transient situations (such as a policy install & connection rematch) that consume enough memory for temporary allocation of swap space to occur.   But in my experience once some swap space is allocated, Gaia/Linux doesn't release it until reboot even if there is now plenty of available memory.  This can be easily seen with commands such as free -m. 

My own personal rule of thumb is that if allocated swap space is more than 5% of total memory (i.e. for 32GB total RAM swap space allocated is 1.6GB) you should probably get some more RAM for your gateway unless a memory leak is present, in which case eventually no amount of RAM will save you.  sk35496: Memory leak detection procedure for a Security Gateway with Gaia OS