fw_full High Memory R80.30

[MattGB7]

Hi Guys,

Over the last 24 hours fw_full is utilising more memory than normal with very little changes made to policy. The only real change to the environment is around 20 additional remote access VPN users.

From my understanding the fw_full process is used for writing log information (I may be in correct so please correct me if I am wrong). The output from top suggest that the process is consuming around 14% of memory which makes it the most memory intensive process on this gateway. As a whole the box is consuming around 78-80% of memory and with the pending addition of additional remote access VPN users coming I wanted to see if there was anything I could do to reduce this.

I have found many articles in relation to high CPU but nothing in relation to high memory utilisation. The gateway is part of a 5200 cluster with both gateways having 8GB of RAM running R80.30 take 155 jumbo.

Any ideas or suggestion welcome. If you need any further information then let me know.

Thanks

Matt

[Timothy_Hall]

Please provide output of command free -m, your box may be using a fair amount of RAM for buffering/caching which is normal but makes memory utilization look much higher than it really is.

[MattGB7]

Hi @Tim See uploaded Image.

Cheers

Matt

[Timothy_Hall]

Approximately 7GB of your 8GB RAM is being used for code execution, but you've only dipped about 1MB into swap so you're OK for now.  fw_full is just a wrapper for the fwd process whose responsibilities are logging on port 257, and acting as a parent process to various security server daemon processes.  It is certainly possible that logging has increased a lot due to all the Remote Access connections, perhaps try just using the standard "Log" for those RA connections instead of Accounting/Detailed/Extended logs? 

Any core dumps in /var/log/dump/usermode?  If a security server process is constantly dying and getting instantly respawned by fw_full over and over again that could account for the issues you are seeing as well.

Also try looking in $FWDIR/log/fwd.elg, any error messages about log buffering issues?  Memory allocation problems?  If logs are not working right or the memory consumption becomes excessive it is possible to kill this process and let cpwd respawn it within 60 seconds, you'll lose logs until it restarts but it won't impact the functionality of the gateway otherwise other than causing a failover.

[MattGB7]

Thskas for the info @Timothy_Hall. 

Logging has been restricted to the stadard\basic log levels for all rules and the additional VPN rules have no logging enabled at the moment. This has reduced memory somewhat however still around 78% utiilsed.

I have attached an extract of the core dumps that are showing in  /var/log/dump/usermode. As you can see nothing from the last 2 weeks.

In terms of $FWDIR/log/fwd.elg all I can rellay see is applicationc ontrol updates being called.

Thanks for the detialed information.

Matt

[Dror_Aharony · Accepted solution]

Please add exact FWD/fw_full memory RES usage (~1.1GB?) & since when FWD has been up:

pidstat -p `pidof fwd` -r 2 3

cpwd_admin list | grep FWD

Please add those here & to Support TAC ticket, if you're sure the FWD's memory usage has risen significantly.