Suddenly experiencing out of memory conditions on security gateways

[Christine_Berns]

We recently started experiencing out of memory errors on our Checkpoint firewall cluster. This is an R77.30 cluster - we realize R77.30 is unsupported and are making plans to migrate to R80.40. In each case, the cluster member that experienced the out of memory condition failed over to the other member, or hung. In each occurrence, /var/log/messages shows the scanengine_b process core dumping and then shortly thereafter, the out of memory condition occurs. Coincidentally, these out of memory conditions first started occurring after the latest threat prevention engine version 59.990000640 was automatically downloaded to our firewalls on May 27. Since then we have had three out of memory occurrences. Prior to that we have never experienced any memory issues in all of the years that we have been running these firewalls. Smartview Monitor graphs show stable memory until suddenly the memory use starts to climb until the out of memory condition occurs.  Attached is output from the /var/log/messages file.  The high CPU usage logged is from CPU-11(FW_0) processing the VPN traffic.  Do the messages in the log file provide any information as to what may be causing the out of memory condition?  If not, can anyone provide some guidance on what we can do to further troubleshoot? 

[PhoneBoy]

The messages confirm it's the Threat Prevention engine that is causing the out of memory issues.
TAC would have to look at the core files to see why it's happening.
Getting on a supported release seems like the next step.

[Christine_Berns]

Would TAC still look at core files from R77.30?  If that's a possibility, I would work with our Checkpoint support vendor to get the files to TAC.

[PhoneBoy]

The Threat Emulation engine is relatively version independent, so it's possible.
However, I can't say for sure.
In any case, the remediation options will be limited to upgrading to a supported release.