Maestro Version Upgrade and Disaster Recovery

[Vanness_Chen]

Hi experts,

Recently, I have been planning an upgrade from R81.10 to R82 for a customer. Since this is my first time performing a Maestro upgrade, I collected system backup files from the production environment and attempted to simulate the upgrade process in a lab environment.

The MHO restore process was successful, but I encountered issues when restoring the Security Gateways (SGs).
After restoring the MHO and adding new SGMs to an SG, the SGMs were able to join successfully. However, they did not receive topology-related configurations (e.g., no management port assignment and therefore no IP address).

To proceed with the restoration, I attempted to use the MHO to transfer the system backup files and Jumbo Hotfix packages to the SGMs and perform a restore. However, when trying to install the Jumbo Hotfix, the system displayed the following warning:

“You are trying to install a package before the First Time Wizard is completed. This can cause the machine to become unstable.”

I am not sure if this is related, but after the restore process, I was unable to log in to the SGM via console.

Additionally, I would like to ask:
If something goes wrong during the upgrade and all SGMs become unavailable, would it still be possible to restore them using system backup, or would a full rebuild be required?

I would greatly appreciate any guidance or shared experience on this topic. Thank you very much.

[emmap]

Is your lab hardware identical to the production SGMs? Not sure why your mgmt port config wouldn't have transferred over, might be something in the $FWDIR/log/blade_config log file on the SGM about that. From that error message about the JHF though it sounds like the SGM didn't build itself into the security group - are you sure you have the R81.10 SP build installed on them? It's a different ISO than the regular R81.10 one.

If there is an issue during the upgrade, the rollback option preferred is to restore the autosnapshot that the upgrade process took. It's documented in the R82 Scalable Platforms Admin Guide in the Maestro Upgrade section.

[Vanness_Chen]

Hi @emmap 

In my lab environment, I only have one MHO_140 and one SGM, so effectively only half of the ports are valid/active (as shown in the screenshot).

In the screenshot, SGM1 and SGM2 are placeholders from the production environment and do not actually exist in the lab. I added SGM3 in the lab for testing.

After completing the restore process, I was unable to log in to the SGM via console.

If restoring from automatic snapshots is the only available method, how can we recover in a scenario where all SGMs have failed?

In what scenarios can SGM system backups be used within a Maestro environment?

[emmap]

The invalid ports being in the security group might not be helping, I see in your log there that you set it to Orch amount 1 but all the Orch 2 ports are still there.

I have seen that issue where it just blanks you back to the login prompt when trying to log in, but I can't remember what it was. This is one SK about it  https://support.checkpoint.com/results/sk/sk180727 - seems there's something in the backup file it doesn't like.

If you just want to test the upgrade process maybe you can skip the restoration. The other challenge you have there though is that with only 1 SGM in your lab you won't see the MVC / fail over part of the upgrade, which is probably the most important part of it. With one SGM the upgrade is just the same as installing a JHF patch.

[Vanness_Chen]

Hi @emmap 

Thank you for providing the documentation.

It seems that if a hardware failure occurs during the upgrade process, the safer approach would be to restore the system to the original version first, and then proceed with the upgrade again using the upgrade package.

[emmap]

Depends on how far you get in the upgrade - if one gateway is upgraded and services are running on it and the second SGM fails, it would be less impact to leave it on the upgraded version. The RMA SGM can then just be added in fresh with R82 installed. 

But yes, whichever way has less impact to the network.