Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

[SecurityNed]

Hello Everyone!

We are currently experiencing very unusual behavior as of writing, wherein when push a policy on our NGFW, at 50% we experience "High Utilization" symptoms wherein traffic gets dropped for a period of time then regains it back right after the policy is successfully pushed. 

This is a photo that represents the behavior above 

And because of the said behavior, after the policy installation, the NGFW also shows to be as disconnected in the SmartConsole, but regains it back after a minute or two.

I have ticked already connection persistence to be at "Keep All Connections" to check whether it improves but unfortunately it didn't and I'm currently lost right now, as just yesterday we didn't experience this behavior at all.

Hoping for someone's insight on this manner.

Thanks!

Ned

EDIT: I'm currently running on R81.10 JHF Take 170 on my firewall

[Chris_Atkinson]

What is the hardware / appliance used and are the Management and Gateway separate machines?

Are you installing both access policy & threat prevention concurrently when the issue occurs?

Would also suggest reviewing sk169096 - Accelerated Install Policy for Access Control Policy to see if the scenario is preventing accelerated policy install.

[SecurityNed]

We're currently experiencing this in a 5200 appliance running on the said version and JHF on the post.

Management and GW are separate machines.

Do take note we also tried changing the values on cpprod_util FwGetParam CP_INSTALL_POLICY_MT_LIMIT to 2, but still experience the same behavior.

You may refer to this video on my attempt in pushing a policy:

[the_rock]

Just about to make breakfast, will check it in a bit.

Cheers,

Andy

[the_rock]

I had that exact issue in my lab while back and after I upgraded to R81.20, it all went away. Ironically enough, to echo what @Chris_Atkinson said, when I would disable accelerated policy install, there was no issue (this was in R81.10)

Andy

[SecurityNed]

Hello @the_rock, 

Unfortunately ticking disabled accelerated policy install still shows the same issue.

Here's the screenshot from the command top when pushing a policy:

Sorry for the late revert as I just came back from a long week out of work.

[the_rock]

No worries, hope you are okay! Health ALWAYS first, always, before anything else. 

Hey, just wondering, have you tried rebooting the fw? If so, did it make any difference?

Andy

[SecurityNed]

Hello @the_rock,

I just did earlier but experienced the same thing after pushing a policy.

[Timothy_Hall]

How do you have Identity Awareness configured on this system?  Two instances of pdpd are consuming gigantic amounts of processing time on your 2 CPUs.  My guess is that you have AD Query configured on a domain with much greater than 250 users; if so you should be using the identity collector software to handle this.  Try unchecking Identity Awareness on your gateway and execute a couple of policy installs, does the problem go away?

[SecurityNed]

Hello @Timothy_Hall, I have both Identity Collector and AD Query enabled for this. But do keep in mind that before we experienced this, at R80.40, we are able to enable both and not encounter the issue. I just tried disabling identity awareness as well, but I still experience the same issue.

[Chris_Atkinson]

Generally both should not be enabled at the same time.

ADQuery is legacy and is replaced by IDC.

[SecurityNed]

Hello @Chris_Atkinson,

But then again, the same behaviour is experienced even after disabling the Identity Awareness blade.

[Chris_Atkinson]

Understand and though it might be a contributing factor I did not state it as the cause for this issue just that it shouldn't be that way.

[Timothy_Hall]

Please provide the outputs of the Super Seven and enabled_blades.  The classic symptom of an overloaded gateway is loss of traffic during a policy push, which is easy to cause on a 2-core system by enabling too many blades while passing a lot of traffic.

[the_rock]

I see what Chris is saying, but in your case, appears that would not be an issue. Hey, just wondering, have you tried maybe disabling IA blade, install policy, re-enable and install again?

Andy

[SecurityNed]

Hello @the_rock,

I have attempted this earlier as what @Timothy_Hall, suggested, but still experienced the same issue even after that.

[the_rock]

Some questions and please forgive me if some of them were answered before.

1) When did this issue originally happen?

2) Any changes done just before the problem started?

3) Any way version can be upgraded to R81.20 jumbo 89 (recommended)?

4) Was there ever TAC case opened for this?

Andy

[SecurityNed]

Hello @the_rock,

No worries, the answers are below:

1) When did this issue originally happen? We happen to notice it right after we upgrade to R81.10. Prior to this on R80.40 everything is working fine

2) Any changes done just before the problem started? The change I think that we did was we attempted to HA the appliance when our RMA arrived, but it failed so we reverted back to, right after that the behavior started.

3) Any way version can be upgraded to R81.20 jumbo 89 (recommended)? If we can push this through, I just want to confirm: can we keep our external firewall at R81.10 for the mean time while we upgrade our Smart-1 appliance and the problematic internal firewall to R82.10?

4) Was there ever TAC case opened for this? So far no, as 1. We need to contact our local distributor first, and they will be the one that will create the ticket on our behalf, which takes a lot of time, that's why as much as possible we're trying to troubleshoot it with the available resources that we have.

[the_rock]

K, thank you for providing those details, that clears some things I was not sure about.

Yes, I would absolutely upgrade to R81.20 if you can and yes, while its not recommended to have cluster members on different versions, I had seen people do it, but personally, I would NOT suggest you do that for more than few days. Maybe best over say one weekend, and possibly monday-tuesday to make sure.

Andy

[Timothy_Hall]

Please provide output of free -m.  My guess is your gateway is short on free memory which can manifest the symptoms you are seeing, since a policy installation is a very memory-intensive action.  If setting "keep all connections" didn't make a difference your issue is probably not CPU-related.  

I assume you just have a single firewall and no cluster?  If a cluster is present does a failover occur during policy install?  Beyond that is there any traffic policing, broadcast suppression on STP events on the switchports the firewall is attached to during a policy installation?