policy verification duration , R80.10

[Matthias_Haas]

Hello all,

we are using R80.10 (MDM and GWs) and for a Policy of roughly 1500 rules it takes 10 minutes on the Mng Server to verify/compile before the Policy is pushed to the GW (which takes about 2 minutes).
This is also true, if we just re-install the Policy (no modification).

From what PhoneBoy mentioned  here :

< In R80+, only the delta is validated, which results in a reduced policy installation time.

I would expect that a re-install does not take that long (as we do not have any delta at all)

So what is your experience in such a case ?

Are 10 minutes still okay for that ?

I know, it depends on a couple of additional parameters, like CPU speed, number of objects ..  

Cheers

Matthias

[Tal_Paz-Fridman]

Hi

We had a case we investigated where a customer was running R80.10 Security Management Server with policies that had more than 3000 rules and hundreds Inline Layers.

Overall Install Policy time was over 20 minutes where the verification process took 13 minutes.

We suggested moving to R80.20 (toady that would be R80.30) since it introduced several improvements in that area.

Using R80.20 the Install Policy time went to down to around 8 minutes and the verification stage took only a few minutes.

With other customers we can see that Policy Verification for a policy of over 3000 rules takes less than a minute using R80.20 or R80.30 

Also check if the machine is not under heavy load from logs. If so I suggest using a dedicated Log Server (MLM).

HTH

Tal

[Timothy_Hall · Accepted solution]

It is true that for the policy verification stage of an Install Policy operation only the deltas are verified, which is because the relatively new multi-threaded java process cpm handles the verification and utilizes multiple cores and the Postgres database to accomplish its task quickly and efficiently. 

All the following assumes that you are spending the vast majority of time waiting for an Install Policy operation when around 40% is displayed by the progress bar in SmartConsole.

Once verification has completed successfully, most of the remainder of the Install Policy process on the SMS side such as Code Generation and Compilation is handled by the single-threaded fwm process which is not directly Postgres-aware, and as such fwm is  working with text-based config files such as objects_5_0.C that were dumped from the database for it to accomplish its task.  This lack of database awareness for fwm is why only one policy install process can be running at a time on an SMS/CMA/Domain.  Every time an install operation is initiated the entire policy must be recompiled by fwm; at one point there was talk about only compiling and pushing the policy deltas to the gateway instead of the whole thing every time, but that never happened to my knowledge.

The best way to help speed this part up is to ensure that the SMS has plenty of RAM and is using zero swap (free -m) and also try to reduce the overall number of objects of the configuration.  The easiest way to do this is from the Object Explorer in SmartConsole by selecting "Unused Objects" in the upper left, and removing unused objects (just make sure that the objects being removed do not have any Automatic NAT rules configured on them).  Removing hundreds of unused objects will reduce the size of the dumped files fwm is having to parse, and make a noticeable difference in overall policy install times. 

I seem to also recall that if the SMS has slow or non-responsive DNS servers configured in the Gaia OS that it could slow down policy install operations, but that may be outdated advice.  However not a bad idea to verify your DNS setup on the SMS and make sure it is working properly.

[Matthias_Haas]

Hi Tim,

thanks a lot, excellent description, very helpfull. Now it´s pretty clear why it still takes that long. I also observed the fact that at around 50% the Progress bar stucks for quite a while but until now I didn´t know why.

Matthias

[Tomer_Noy]

I can add a bit more info (based on some knowledge of the internals).

In R80.x, the new cpm process & Postgres have indeed given us new capabilities and the ability to handle many operations using multiple threads. Much of this cannot be applied to logic that still runs under fwm.

Fortunately, the smart people who worked on policy installation in R80.10, created a mechanism that will perform the policy verification only on the delta of changes. The way that this is done, is that cpm "dumps" the full policy for fwm, but also outputs the IDs of rules that have actually changed since the last policy installation. Then, the policy verification code takes that into account to optimize the policy verification (mainly the rule-hide-rule calculation).

Due to this mechanism, the first policy verification (after import or upgrade) may take some time, but the next one should be much faster.

In addition, we are working on another improvement for R80.40 that will make the verification significantly faster (especially with large rulebases), even on the first run.

Question: Did you happen to notice if there is a difference between your first time and the subsequent attempts?

Also, 10 minutes is quite a long time for a policy of 1500 rules. If this didn't improve, I would suggest opening a TAC request and they can help you check the environment to see if there is a reason for this (possibly other load or HW).