Policy installation failed on gateway (error code: 0-3-2000172-1)

[imamuzic]

Hello,

We recently ran on the problem with the policy installation with an error code from the subject and I would like to write here steps for successeful resolution as it might help someone else.

First of all , there is already an SK for the problem (https://support.checkpoint.com/results/sk/sk159952) but it writes only about Snort custom IPS Protections which we didn't had so this SR was not relevant to us, so we opened SR and the support engineer provided as with a fix procedure, but the procedure wasn't complete, so it didn't worked for us, but we got a hint where the problem might be.  So here is the complete procedure I figured out to be working finally:

1. The root cause was apparently a faulty IPS signature update that landed on the firewall cluster. Since ZD2 was an active cluster member at the time of the policy install, the bug affected only that firewall, but the policy was unable to install onto another cluster member as well due to „For gateway clusters, if installation on a cluster member fails, do not install on that cluster“ option was set on the management server (default setting).
2. I thought that the faulty cluster member (‘ZD2’) reboot might help, but after rebooting the cluster member was unavailable for HTTPS/SSH access since it could not install/fetch the policy because of the root cause and so it had to revert to default policy that was preventing us to access the gateway’s management interface from the network. Also, since the gateway was unable to load the production policy it was fallen out from the cluster, so we temporary lost redundancy as well. That policy was removed with the “fw unloadlocal” command (the serial console connection was needed), so HTTPS/SSH connectivity was restored.
3. The next suspicion was on the hard drive fault, but the ‘diagMain’ tool’s “Long Disk Test” wasn’t revealed any issues with the hard drive.
4. The SR “SR#6-0003844926 policy installation failure with the error code: 0-3-2000172-1” was created asking Check Point TAC for support.
5. The support engineer responded with the procedure that was to revert to one IPS signature version back, but that didn’t helped, but then we checked the Install Policy logs for last successful policy install which was 1.2.2024@14:47, so we decided to revert back to a IPS signature version older then the last policy successful install which was the one from 31.2.21024 and after that we switched to latest IPS signature update (04.02.2024) and we finally managed to recover the install policy process and resolve the issue.
6. Recovery steps which I’m writing here since :
1. Important: DO NOT REBOOT a faulty cluster member, because you will lose management connectivity to the member since fetching the policy upon reboot will fail and resulting with default policy installation which will prevent access to the management interface for HTTPS/SSHand serial console connectivity will be needed to restore it. Also, this gateway will not be a member of the cluster anymore due to inability to fetch the production policy that contains cluster configuration.
2. CREATE A SNAPSHOT OF THE MANAGEMENT SERVER!!! :Gaia (HTTPS portal to the management server): Maintenance -> Snaphost Management
3. Smart Console -> Gateway Cluster Properties -> IPS -> IPS Update Policy -> temporary change to “Use IPS management updates”.
4. Smart Console -> Recent Tasks -> find out the last successful Policy Install date for the cluster and note it.
5. Smart Console -> Security Policies -> Threat Prevention -> Custom Policy -> Updates -> IPS -> Update Now (click on the arrow) -> Switch to version…-> choose version older than the last successful policy update you found out from the previous step -> Switch -> wait for the task to complete.
6. Install policy on the cluster – it should be successful now.
7. Repeat step 3, but this time choose the latest IPS version, switch to it and wait for the task to complete.
8. Install policy on the cluster – it should be successful now and the main issue resolved.

Regards,

Igor

[_Val_ · Accepted solution]

Thanks for sharing, great one!

[the_rock]

Hey Igor,

Fantastic post, thats SUPER HELPFUL mate. 

👍✌️

Best,

Andy

[DavidG]

We just saw something different, adding here for awareness.  The issue was one device in the cluster getting messed up when enabling NGTP blades, they were removed but the problem device didn't uninstall the blades or TP policy, presumably due to an issue mid-policy install:

• Check on each cluster member which blades are enabled either using the command 'enabled_blades' or via cpview
• If the problem device has a different number of blades, incorrect policy or is generally not getting the right configuration from SmartConsole then reset SIC - this will remove all the blades and policies so you can start from fresh.
• For some reason in our case the problem member didn't default to the correct initial policy after a cpconfig>SIC reset and an 'fw unloadlocal' was needed to allow SmartConsole to re-initialize SIC; therefore be prepared to have LoM or console access 

After the SIC reset the error was resolved.

[the_rock]

I will just add to this, because I realize that lots of people usually may not even think about it, but here it comes:

-if you do cprestart on the gateway, cpstop will remove the policy temporarily, similar to doing fw unloadlocal and cpstart will put it back

-if you do sic reset, it will load initial policy

Something to be aware of.

Andy