Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
SecurityNed
Collaborator

Checkpoint NGFW Showing "High Utilization" Symptoms When Pushing Policies

Hello Everyone!

We are currently experiencing very unusual behavior as of writing, wherein when push a policy on our NGFW, at 50% we experience "High Utilization" symptoms wherein traffic gets dropped for a period of time then regains it back right after the policy is successfully pushed. 

This is a photo that represents the behavior above 

Image (1).jpg

And because of the said behavior, after the policy installation, the NGFW also shows to be as disconnected in the SmartConsole, but regains it back after a minute or two.

I have ticked already connection persistence to be at "Keep All Connections" to check whether it improves but unfortunately it didn't and I'm currently lost right now, as just yesterday we didn't experience this behavior at all.

Hoping for someone's insight on this manner.

Thanks!

Ned

EDIT: I'm currently running on R81.10 JHF Take 170 on my firewall

19 Replies
Chris_Atkinson
Employee Employee
Employee

What is the hardware / appliance used and are the Management and Gateway separate machines?

Are you installing both access policy & threat prevention concurrently when the issue occurs?

Would also suggest reviewing sk169096 - Accelerated Install Policy for Access Control Policy to see if the scenario is preventing accelerated policy install.

 

CCSM R77/R80/ELITE
SecurityNed
Collaborator

We're currently experiencing this in a 5200 appliance running on the said version and JHF on the post.

Management and GW are separate machines.

Do take note we also tried changing the values on cpprod_util FwGetParam CP_INSTALL_POLICY_MT_LIMIT to 2, but still experience the same behavior.

You may refer to this video on my attempt in pushing a policy:

vid1.mp4
Video Player is loading.
Current Time 0:00
Duration 0:00
Loaded: 0%
Stream Type LIVE
Remaining Time 0:00
 
1x
    • Chapters
    • descriptions off, selected
    • captions off, selected
      (view in My Videos)

      the_rock
      Legend
      Legend

      Just about to make breakfast, will check it in a bit.

      Cheers,

      Andy

      the_rock
      Legend
      Legend

      I had that exact issue in my lab while back and after I upgraded to R81.20, it all went away. Ironically enough, to echo what @Chris_Atkinson said, when I would disable accelerated policy install, there was no issue (this was in R81.10)

      Andy

      SecurityNed
      Collaborator

      Hello @the_rock

      Unfortunately ticking disabled accelerated policy install still shows the same issue.

      Here's the screenshot from the command top when pushing a policy:

      top.png

      Sorry for the late revert as I just came back from a long week out of work.

      the_rock
      Legend
      Legend

      No worries, hope you are okay! Health ALWAYS first, always, before anything else. 

      Hey, just wondering, have you tried rebooting the fw? If so, did it make any difference?

      Andy

      SecurityNed
      Collaborator

      Hello @the_rock,

      I just did earlier but experienced the same thing after pushing a policy.

      Timothy_Hall
      Legend Legend
      Legend

      How do you have Identity Awareness configured on this system?  Two instances of pdpd are consuming gigantic amounts of processing time on your 2 CPUs.  My guess is that you have AD Query configured on a domain with much greater than 250 users; if so you should be using the identity collector software to handle this.  Try unchecking Identity Awareness on your gateway and execute a couple of policy installs, does the problem go away?

      Gateway Performance Optimization R81.20 Course
      now available at maxpowerfirewalls.com
      SecurityNed
      Collaborator

      Hello @Timothy_Hall, I have both Identity Collector and AD Query enabled for this. But do keep in mind that before we experienced this, at R80.40, we are able to enable both and not encounter the issue. I just tried disabling identity awareness as well, but I still experience the same issue.

      Chris_Atkinson
      Employee Employee
      Employee

      Generally both should not be enabled at the same time.

      ADQuery is legacy and is replaced by IDC.

      CCSM R77/R80/ELITE
      SecurityNed
      Collaborator

      Hello @Chris_Atkinson,

      But then again, the same behaviour is experienced even after disabling the Identity Awareness blade.

      Chris_Atkinson
      Employee Employee
      Employee

      Understand and though it might be a contributing factor I did not state it as the cause for this issue just that it shouldn't be that way.

      CCSM R77/R80/ELITE
      Timothy_Hall
      Legend Legend
      Legend

      Please provide the outputs of the Super Seven and enabled_blades.  The classic symptom of an overloaded gateway is loss of traffic during a policy push, which is easy to cause on a 2-core system by enabling too many blades while passing a lot of traffic.

      Gateway Performance Optimization R81.20 Course
      now available at maxpowerfirewalls.com
      the_rock
      Legend
      Legend

      I see what Chris is saying, but in your case, appears that would not be an issue. Hey, just wondering, have you tried maybe disabling IA blade, install policy, re-enable and install again?

      Andy

      SecurityNed
      Collaborator

      Hello @the_rock,

      I have attempted this earlier as what @Timothy_Hall, suggested, but still experienced the same issue even after that.


      the_rock
      Legend
      Legend

      Some questions and please forgive me if some of them were answered before.

      1) When did this issue originally happen?

      2) Any changes done just before the problem started?

      3) Any way version can be upgraded to R81.20 jumbo 89 (recommended)?

      4) Was there ever TAC case opened for this?

      Andy

      SecurityNed
      Collaborator

      Hello @the_rock,

      No worries, the answers are below:

      1) When did this issue originally happen? We happen to notice it right after we upgrade to R81.10. Prior to this on R80.40 everything is working fine

      2) Any changes done just before the problem started? The change I think that we did was we attempted to HA the appliance when our RMA arrived, but it failed so we reverted back to, right after that the behavior started.

      3) Any way version can be upgraded to R81.20 jumbo 89 (recommended)? If we can push this through, I just want to confirm: can we keep our external firewall at R81.10 for the mean time while we upgrade our Smart-1 appliance and the problematic internal firewall to R82.10?

      4) Was there ever TAC case opened for this? So far no, as 1. We need to contact our local distributor first, and they will be the one that will create the ticket on our behalf, which takes a lot of time, that's why as much as possible we're trying to troubleshoot it with the available resources that we have.

      the_rock
      Legend
      Legend

      K, thank you for providing those details, that clears some things I was not sure about.

      Yes, I would absolutely upgrade to R81.20 if you can and yes, while its not recommended to have cluster members on different versions, I had seen people do it, but personally, I would NOT suggest you do that for more than few days. Maybe best over say one weekend, and possibly monday-tuesday to make sure.

      Andy

      Timothy_Hall
      Legend Legend
      Legend

      Please provide output of free -m.  My guess is your gateway is short on free memory which can manifest the symptoms you are seeing, since a policy installation is a very memory-intensive action.  If setting "keep all connections" didn't make a difference your issue is probably not CPU-related.  

      I assume you just have a single firewall and no cluster?  If a cluster is present does a failover occur during policy install?  Beyond that is there any traffic policing, broadcast suppression on STP events on the switchports the firewall is attached to during a policy installation?

      Gateway Performance Optimization R81.20 Course
      now available at maxpowerfirewalls.com

      Leaderboard

      Epsum factorial non deposit quid pro quo hic escorol.

      Upcoming Events

        CheckMates Events