Why checkpoint cluster suddenly freezed ?

[Prashan_Attanay]

We have checkpoint cluster, both worked fine. But suddenly one device in cluster suddenly freezes. We had to manually restart the freeze device to activate the other device as active. Even checkpoint support engineer doesn't found root cause to freeze. 

I want to know why sudden freeze happen in cluster ? 

Please find the log details as well 

[fw4_1];FW-1: [cul_load_freeze][CUL - Cluster] CUL should be OFF (short timeout of 10 seconds expired) but at least one member reported high CPU usage 9 seconds ago 

kernel: [fw4_1];FW-1: [cul_load_freeze][CUL - Cluster] CUL_CLUSTER is FREEZE_ON and should stay ON, updating time, kernel CPU usage [85%] 

[Daniel_Taney]

This message occurs when you have a Cluster member under heavy CPU load. Have you changed anything recently that could cause an increase in CPU load? (i.e. enabled a new Software Blade, added new IPS protections, enabled HTTPS inspection?) 

Depending on what version you are running on the Gateway, you may be able to use cpview (sk101878) to look at the history on the Gateway to see what caused the CPU spikes. If you can identify what process is causing the spikes, you may be able to work with TAC to adjust / tune CoreXL and SecureXL to provide better efficiency on your Gateway. 

I personally saw this happen when a number of "Critical" CPU level IPS signatures inadvertently got enabled. 

Hope this helps! Good luck!

[Prashan_Attanay]

Hi Daniel,

Thank you for your reply.

We are running 77.30 version. Yes we did update IPS. Same day that occurs cluster crashed also. TAC suggest us to install the latest Jumbo Hotfix. 

By the can you guide me how to get history of CPU spikes

[Daniel_Taney]

I would start with sk101878. Hopefully, your cluster already has CPView History enabled. Without history enabled, you won't be able to go back in time and see what happened. 

cpview history stat will show you whether the feature is enabled. If it is, you should be able to scroll back through the CPView data. If it isn't, cpview history on will enable the feature. Unfortunately, you'll be stuck waiting to see if it happens again if it wasn't turned on.

Once you establish whether it is enabled or not, I'd check out the CPView User Guide. For some reason, I can only find one designated for R77.20. I don't know if there is a newer guide out there somewhere that I'm just not finding? Regardless, the guide should give you a pretty good explanation on how to use the tool to extract the information you're looking for.

Good luck!

[Prashan_Attanay]

Thank you Daniel. I refer the docs.