topic Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies in Firewall and Security Management

Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Fri, 29 Nov 2024 04:11:27 GMT

Hello Everyone!

We are currently experiencing very unusual behavior as of writing, wherein when push a policy on our NGFW, at 50% we experience "High Utilization" symptoms wherein traffic gets dropped for a period of time then regains it back right after the policy is successfully pushed.

This is a photo that represents the behavior above

And because of the said behavior, after the policy installation, the NGFW also shows to be as disconnected in the SmartConsole, but regains it back after a minute or two.

I have ticked already connection persistence to be at "Keep All Connections" to check whether it improves but unfortunately it didn't and I'm currently lost right now, as just yesterday we didn't experience this behavior at all.

Hoping for someone's insight on this manner.

Thanks!

Ned

EDIT: I'm currently running on R81.10 JHF Take 170 on my firewall

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

Chris_Atkinson — Fri, 29 Nov 2024 10:37:11 GMT

What is the hardware / appliance used and are the Management and Gateway separate machines?

Are you installing both access policy & threat prevention concurrently when the issue occurs?

Would also suggest reviewing sk169096 - Accelerated Install Policy for Access Control Policy to see if the scenario is preventing accelerated policy install.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

the_rock — Fri, 29 Nov 2024 18:20:24 GMT

I had that exact issue in my lab while back and after I upgraded to R81.20, it all went away. Ironically enough, to echo what @Chris_Atkinson said, when I would disable accelerated policy install, there was no issue (this was in R81.10)

Andy

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

Timothy_Hall — Fri, 29 Nov 2024 18:46:53 GMT

Please provide output of free -m. My guess is your gateway is short on free memory which can manifest the symptoms you are seeing, since a policy installation is a very memory-intensive action. If setting "keep all connections" didn't make a difference your issue is probably not CPU-related.

I assume you just have a single firewall and no cluster? If a cluster is present does a failover occur during policy install? Beyond that is there any traffic policing, broadcast suppression on STP events on the switchports the firewall is attached to during a policy installation?

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Mon, 16 Dec 2024 12:12:40 GMT

Hello @the_rock,

Unfortunately ticking disabled accelerated policy install still shows the same issue.

Here's the screenshot from the command top when pushing a policy:

Sorry for the late revert as I just came back from a long week out of work.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

the_rock — Mon, 16 Dec 2024 12:39:36 GMT

No worries, hope you are okay! Health ALWAYS first, always, before anything else.

Hey, just wondering, have you tried rebooting the fw? If so, did it make any difference?

Andy

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Mon, 16 Dec 2024 12:46:00 GMT

We're currently experiencing this in a 5200 appliance running on the said version and JHF on the post.

Management and GW are separate machines.

Do take note we also tried changing the values on cpprod_util FwGetParam CP_INSTALL_POLICY_MT_LIMIT to 2, but still experience the same behavior.

You may refer to this video on my attempt in pushing a policy:

(view in My Videos)

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

the_rock — Mon, 16 Dec 2024 12:47:56 GMT

Just about to make breakfast, will check it in a bit.

Cheers,

Andy

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

Timothy_Hall — Mon, 16 Dec 2024 12:59:38 GMT

How do you have Identity Awareness configured on this system? Two instances of pdpd are consuming gigantic amounts of processing time on your 2 CPUs. My guess is that you have AD Query configured on a domain with much greater than 250 users; if so you should be using the identity collector software to handle this. Try unchecking Identity Awareness on your gateway and execute a couple of policy installs, does the problem go away?

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Mon, 16 Dec 2024 13:18:44 GMT

Hello @Timothy_Hall, I have both Identity Collector and AD Query enabled for this. But do keep in mind that before we experienced this, at R80.40, we are able to enable both and not encounter the issue. I just tried disabling identity awareness as well, but I still experience the same issue.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Mon, 16 Dec 2024 13:19:18 GMT

Hello @the_rock,

I just did earlier but experienced the same thing after pushing a policy.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

Chris_Atkinson — Mon, 16 Dec 2024 13:26:41 GMT

Generally both should not be enabled at the same time.

ADQuery is legacy and is replaced by IDC.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Mon, 16 Dec 2024 13:28:30 GMT

Hello @Chris_Atkinson,

But then again, the same behaviour is experienced even after disabling the Identity Awareness blade.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

Chris_Atkinson — Mon, 16 Dec 2024 14:52:58 GMT

Understand and though it might be a contributing factor I did not state it as the cause for this issue just that it shouldn't be that way.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

Timothy_Hall — Mon, 16 Dec 2024 13:44:15 GMT

Please provide the outputs of the Super Seven and enabled_blades. The classic symptom of an overloaded gateway is loss of traffic during a policy push, which is easy to cause on a 2-core system by enabling too many blades while passing a lot of traffic.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

the_rock — Mon, 16 Dec 2024 13:46:08 GMT

I see what Chris is saying, but in your case, appears that would not be an issue. Hey, just wondering, have you tried maybe disabling IA blade, install policy, re-enable and install again?

Andy

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Mon, 16 Dec 2024 13:48:39 GMT

Hello @the_rock,

I have attempted this earlier as what @Timothy_Hall, suggested, but still experienced the same issue even after that.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

the_rock — Mon, 16 Dec 2024 14:49:36 GMT

Some questions and please forgive me if some of them were answered before.

1) When did this issue originally happen?

2) Any changes done just before the problem started?

3) Any way version can be upgraded to R81.20 jumbo 89 (recommended)?

4) Was there ever TAC case opened for this?

Andy

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

SecurityNed — Tue, 17 Dec 2024 06:45:22 GMT

Hello @the_rock,

No worries, the answers are below:

1) When did this issue originally happen? We happen to notice it right after we upgrade to R81.10. Prior to this on R80.40 everything is working fine

2) Any changes done just before the problem started? The change I think that we did was we attempted to HA the appliance when our RMA arrived, but it failed so we reverted back to, right after that the behavior started.

3) Any way version can be upgraded to R81.20 jumbo 89 (recommended)? If we can push this through, I just want to confirm: can we keep our external firewall at R81.10 for the mean time while we upgrade our Smart-1 appliance and the problematic internal firewall to R82.10?

4) Was there ever TAC case opened for this? So far no, as 1. We need to contact our local distributor first, and they will be the one that will create the ticket on our behalf, which takes a lot of time, that's why as much as possible we're trying to troubleshoot it with the available resources that we have.

Re: Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

the_rock — Tue, 17 Dec 2024 12:27:05 GMT

K, thank you for providing those details, that clears some things I was not sure about.

Yes, I would absolutely upgrade to R81.20 if you can and yes, while its not recommended to have cluster members on different versions, I had seen people do it, but personally, I would NOT suggest you do that for more than few days. Maybe best over say one weekend, and possibly monday-tuesday to make sure.

Andy