How a recovered VSX cluster member obtains the Security Policy

[MartinOles]

Hi,

Could you please clarify to me, how security policy installation or fetching policy works on VSX and MultiDomain server.

We do have VSX cluster, two members, somebody did changes on one of rulebase for one of virtual system and never pushed (installed) policy, changes were only published, policy is assigned to correct virtual system. After while we did cluster member reboot. Such went according to expectation with small issue with sync network which recovered after quite long time (15 minutes?, issue on other vendor devices).

After reboot now we do have policy on one of virtual system included changes, which were not pushed (installed) by administrator. And I am a bit puzzled. According to sk119332:

"Security policy changes must be pushed to the Security Gateway before they will be implemented by an "fw fetch" command. The "fw fetch" compares the compiled policy on the Security Management server with the latest policy on the Security Gateway."

From it I do understand, that if policy changes were only published, they were actually not compiled and therefore could not be fetched during cluster member boot.

But in "ClusterXL AdminGuide R80.40" is written:

"When a failed cluster member recovers, first it tries to fetch a policy from one of the peer Active Cluster
Members. The assumption is that the other Cluster Members have a more up to date policy. If fetching a
policy from peer cluster member fails, the recovered cluster member compares its own local policy to the
policy on its Management Server. If the policy on the Management Server is more up to date than the one on
the recovered cluster member, the policy is fetched from the Management Server. "

Such might be according to our observation, that during boot, and issue with sync network, cluster member fetch policy from the Management Server including all published changes.

Could you please give me hint, where is policy fetch maybe better described for scenario where cluster member is recovering?

I would like to have a bit deeper knowledge of this behavior for the future, as is in this particular case an administrator messed up that four months ago assigned Global policy with "drop" to affected virtual system policy and did not pushed policy, so traffic worked, but after cluster members reboot we were facing sudden traffic failure due to policy drop.

Thank you.

[_Val_]

Both SKs provide you with 100% accurate information.

Here are some points:

1. Before any policy can be fetched, you need to compile it on the management server side. Compilation creates a policy package files set which is 

1. stored on the MGMT server side in a folder belonging to the GW/Cluster
2. sent to that GW or cluster where, before the final stage of the installation happens, it is once again stored locally

2. When a cluster member boots up, during the initialization process, GW checks

1. there is already a policy package stored locally, (p. 1b)
2. other cluster members have the same or newer policy installed
3. management has a newer policy compiled and available for the external fetch

3. Depending on the results of those checks, the GW in question loads the local policy or fetches a newer one from another source.

The main requirement here is, that the policy should be up to date and also equal to the policy installed on another cluster member.

All the above is for both VSX and physical clusters. However, with VSX there is another part of the configuration requiring checks and fetches, and it happens even before applying the security policy. I am talking about the VSX provisioning scripts, which define who VSs are built and configured. 

The logic is very similar: verify what/if you have it locally, check with MGMT and the other cluster members that the scripts are up to date, fetch and apply the newest, if required, or use the local ones.

Concerning the question in your last paragraph, the situation when a "bad" policy compilation results are available on the management but not on the GWs is only possible when you try to install a policy from your MGMT server, and the compilation passes, but the installation fails because GWs are not accessible for whatever reason. 

In this case, yes, the new cluster member will have to fetch the latest bad version, but you don't know it is bad because you never succeeded in installing it.

Saving database or publishing changes does not cause this situation, because it does not have a compilation stage. In fact, you cannot compile policy outside of the installation operation. 

Hope this makes sense, if not, let me know, and I will elaborate.

You can also look into this recorded session, I describe the policy installation process there:

https://community.checkpoint.com/t5/DACH-Germany-Austria-and/Troubleshooting-Series-Part-1-Video-and...

Mind you might need to join the group to access it.