slow policy install over T1

[Kevin_Orrison]

I'm not sure if there is anything else that can be done, but I wanted to see if anyone had any suggestions. I am testing a remote firewall centrally manged on the other side of a T1 circuit. Currently, it takes 30 minutes, sometime longer, for policy to install. I am wondering if there is anything I can do on the Check Point end to make sure everything is optimized. Maybe it would help to know exactly what is being transferred during policy install. Also, this link doesn't seem to be very highly utilized.

[PhoneBoy]

In versions of management prior to R80, the biggest contributor to long policy installation times is the policy verification process with thousands of rules.

This is because each rule is validated against every other rule each time the policy is compiled and installed.

In R80+, only the delta is validated, which results in a reduced policy installation time.

Once the policy is validated, the policy is pushed to the gateway and installed.

The exact mechanism is different from R77.x to R80, but it's still effectively the whole policy being pushed.

In later releases, we plan to push only the delta (note: this will require gateways to be upgraded).

If you want to reduce the policy installation time:

• Upgrade management to R80.10+
• Push policies with fewer rules

[Timothy_Hall]

Cleaning up unused objects may help lower installation time in this case as well, since all objects are sent as part of a policy installation to a gateway whether they are directly referenced in that gateway's security policy or not.  In R80+ management, you can find unused objects by bringing up the Object Explorer and in the upper-left of the window selecting "Unused Objects" instead of "All".  In R77.30 management Unused Objects can be viewed from the Search...Query Network Objects...Refined Filter...Refine By menu screen.  However before deleting what appears to be an unused object, make sure it does not have Automatic NAT enabled on its NAT tab.  Deleting an object configured with Automatic NAT on it can definitely break stuff.

For further optimization opportunities, take a look in directory $FWDIR/state/__tmp/FW1 on the gateway.  Here you can see all the files sent from the SMS to the gateway during a policy installation, any big files here will definitely slow things down over a T1.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Kevin_Orrison]

I am aware of the added delta benefits of R80+ and I am definitely looking forward to upgrading from R77.30 in the next couple months. Verification only takes a couple minutes. The bulk of the slowness is after verification during the actual policy install to the gateway. Just to be clear, if I clean up unused objects, will that only improve the verification time or the actual policy install to the gateway?

[Daniel_Taney]

Do you have any visibility into the circuit itself (latency, utilization, packet loss, etc)? Is the circuit highly utilized at the times when you push policy? The Policy Installation itself may not be consuming the full size of the pipe, but other things could be. 

Cleaning up unused objects would help reduce the overall size of the policies. Unless you have lots and lots of objects, this may not make an appreciable difference, though.

[Kevin_Orrison]

The circuit hasn't been the most reliable but, no there doesn't appear to be high util during policy install. I have alerts that would also notify me. I am going do some object clean up and see what that does.

[Timothy_Hall]

After verification is complete CPTA (Check Point Transfer Agent - cpd) is invoked to send the policy to the gateway.  This is probably where you are waiting the longest due to the limited T1 bandwidth; reducing the number of objects will help slightly here.  A good test would be to run top on both the SMS and gateway during a policy install.  You should see some CPU utilization on the SMS as it performs the verification/compilation, then it will drop back down.  CPTA is running at this point, and during this period you should see negligible CPU usage on both the SMS and gateway.

Once CPTA finishes its job you will see a CPU spike on the gateway as it does the atomic load and connection rematching.  You should be able to measure the time between the CPU spike on the SMS and gateway to figure out how much time is being spent in CPTA.  It is also possible that the atomic load and connection rematching is taking awhile on the gateway, and you can measure this from the start of the CPU spike on the gateway to when it drops back down and the SMS reports policy installation is complete.  What model is the remote gateway?

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Kevin_Orrison]

3200 cluster

[Timothy_Hall]

The 3200 is a quite reasonable box, it shouldn't take that long for the atomic load & rematch.  Pretty sure you'll find CPTA is where all the time is being spent.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com