About upgrade a smart-1 of Management High Availability

[TSOL]

Hello Team

We want to update two management servers configured for Management High Availability from R81.10 to R81.20.

If I upgrade in the order of "primary management server" followed by "secondary management server," will the issue described in sk179794 always occur?

Upgrading Security Management Servers in Management High Availability from R80.20 and higher (checkp...

Thank you for the advice.

[AkosBakos]

What do you mean under "issue" 

This is a simple upgrade guide.

Akos

[TSOL]

Thank you for the reply.

This SK"Upgrade fails on Secondary Management Server due to SIC Communication issue (checkpoint.com)" documents mention that after updating the primary management server, the secondary management server cannot be updated, which is making us hesitant to proceed.

[AkosBakos · Accepted solution]

Hi @TSOL 

in the SK the cause is the following:

The Secondary Management Server fails to communicate with / get a response from the Primary Management Server to check that the Primary Management Server was already upgrade

As soluiton, there are basic troubleshooting steps.

From my point of view, I had two MGMT HA updrade in the last 365 days. I didn't run into this issue, but it does not mean that you won't meet this problem.

You won't know it, until you don't do/try it 🙂

Worst scenario, you revert from checkPoint snapshot.

Cheers

A

[TSOL]

Hey Akos

Thank you for sharing your experience.

Since it seems like it's an issue that rarely occurs, there’s probably no need to worry too much, so I’ll proceed with the update.

[the_rock]

Hey @TSOL ,

I would follow link below for official upgrade steps.

Best,

Andy

https://sc1.checkpoint.com/documents/R81.10/WebAdminGuides/EN/CP_R81.10_Installation_and_Upgrade_Gui...

[the_rock]

orry, I see you referred to same link as one I posted, apologies. And, yes, as @AkosBakos had said, you never know if issue would happen until you try it. I tested same process in the lab, worked fine. Think about it this way...

Say you are upgrading just one mgmt and say single gateway managed by it, from, ie R80.40 to R81.20. Technically, since thats major upgrade, it may load initial policy on the fw after reboot and then you need to install right policy and all works fine again.

Though, in your case, since its minor upgrade, I would be really surprised if you had SIC issue...I had done this in lab few times for mgmt HA, never had an issue.

One time, I did not even follow the steps, I simply went by logic as cluster HA, I upgraded standby mgmt member, which was also secondary, rebooted, then did active (which was primary) and all was fine. 

Best,

Andy

[TSOL]

Thank you for sharing your experience.

If the issue doesn't occur after several attempts, I believe there's no need to worry too much, so I'd like to proceed with the update in the order of "primary" to "secondary".

[emmap]

Just a tip in case you're not aware, don't install the JHF on the primary until the secondary has completed its version upgrade. As part of its upgrade the secondary will attempt a full sync of the database, which will fail (leading to an automatic rollback) if the primary has the JHF installed, as the secondary won't at that stage. So do the version upgrade on both, make sure they're both happy, then proceed with JHF install on both. 

Using the blink package is not a workaround here as it's not supported to upgrade a secondary management server with the blink package.

[JozkoMrkvicka]

No plans to address this nonsense "feature" ? At least to block upgrade of Secondary management if Primary was upgraded/downgraded while JHF was installed on Primary management.

Some customers want to test full functionality of upgraded management before doing upgrade of HA member. How they are supposed to test functionality (with latest recommended JHF) if they are forced to upgrade Secondary management first ?

It is even instructed by Check Point to install latest Recommended JHF on every version.

[emmap]

I understand it's not ideal, but I don't know if there's anything we can do about blocking sync between Mgmt HA when they are not on the same JHF take. If the JHF has affected the database and we sync something to a server without that JHF then we could be causing much greater issues. Customers with specific scenarios to test can either do so in a lab if that's feasible, or do a test upgrade of the primary, run through test scenarios, then revert back to the snapshot taken before the upgrade procedure and then run the full upgrade later. Either way, the upgrade procedure always takes an upgrade that is available for rollback if required. 

Alternatively the secondary management server can be clean installed, patched and re-SIC'd after a full upgrade of the primary. 

[TSOL]

Thank you for this information.
Based on what you mentioned, would the sequence be as follows?

*My Smart-1 is on R81.10 take 335:

1. Use the R81.20 Gaia fresh install and upgrade to upgrade the primary management server.
2. Use the R81.20 Gaia fresh install and upgrade to upgrade the secondary management server.
3. Use the R81.20 Security Management JHF 176 to upgrade the primary management server.
4. Use the R81.20 Security Management JHF 176 to upgrade the secondary management server.

I appreciate all the advice.

[AkosBakos]

Hi @TSOL 

Don't mix it.

The Blink Package (image) contains the jumbo hotfix, in this case the take 76. 

The Major Verison contains only the R81.20 main release without jumbo hotfix.

Nowadays I use Blink image for upgrading Check Point products  if it is allowed.

Follow the steps in the R81.20 Installation and Upgrade guide start with the backups

Akos

[TSOL]

Hi Akos

Understood.
I will first apply the R81.20 blink package to the primary management server, and once completed,

I will apply it to the secondary management server.

[emmap]

You can't use the blink packages, it won't work on the secondary.

Apply the R81.20 Gaia fresh install and upgrade to the primary, then to the secondary, then under the hotfixes section in the WebUI you'll see JHF T76 available. Once the version upgrade is done on both, apply that to the primary and secondary. 

[TSOL]

Hi emmap,

Thank you fot the reply.

Summarizing the above, does it look like this?

My Smart-1 is on R81.10 take 335:

1.Use the R81.20 Gaia fresh install and upgrade to upgrade the primary management server.
2.Use the R81.20 Gaia fresh install and upgrade to upgrade the secondary management server.
3.Use the JHF 176 to upgrade the primary management server. "Not blink package"
4.Use the JHF 176 to upgrade the secondary management server. "Not blink package"

Thank you for the advice.

[JozkoMrkvicka]

These steps are correct for the upgrade of management (SMS or MDS).

But you missed couple of steps which are needed to be done BEFORE upgrade. Like do snapshots, save content of all manually modified inspect files (.def), update CPUSE deployment agent, update latest upgrade package for target version, run PUV, fix all errors and examine warnings.

All of these steps are mentioned in admin guide which you posted in your very first post.

[AkosBakos]

Hi @emmap 

Hm, interesting, on primary management the blink image works. So the secondary differ from primary?

Akos

[emmap]

Last time I checked that was the case.

[the_rock]

I used it 3 times before in the lab and it worked fine, on actual secondary server as well. I never actually verified in the documentation if its supported, but definitely worked for me. 

Andy

[AkosBakos]

I asked around among my colleauages about this.

I got only this restriction:

You should upgrade/patch exactly the same way the primary and secondary MGMT (it is in the CCSM study guide)

It could be tricky. Tthink about it:  the customer wants a secondary MGMT after the one and only MGMT has been working for 3 years..... 🙂

A

[the_rock]

I have R81.20 mgmt ha, primary is on jumbo 79, secondary does not have any installed, no issues.

R82 lab, primary got no jumbo, secondary jumbo 40, works fine.

Is it officially supported? I will take an educated guess and say probably not, but it works 100%. Do I encourage anyone to do it this way? I do not, but in my lab, functions without any problem.

Andy

[AkosBakos]

Not supported, but should work 🙂

[the_rock]

Makes sense.

[emmap]

If you kick off a full sync of the Mgmt servers, does it work?

[the_rock]

Absolutely.

[emmap]

Last time I tried it failed, but that was a while ago - looks like I need to try again.

[the_rock]

Agree 100% @JozkoMrkvicka , you bring up an EXCELLENT point mate. Its certainly something that should be changed/fixed.

Andy

[Bob_Zimmerman]

I would just do a clean install on the secondary, configure it as a secondary management, install whatever jumbo, then synchronize it with the primary. Same process you would use to replace a failed secondary. It's simple, and it completely avoids all of the concerns in this thread.

[JozkoMrkvicka]

Instead clean install I would rather do full re-image (from USB/LOM) of secondary management to change file system from EXT3 to XFS and also fix issue with partition alignment to 1024-byte boundaries, if SSDs are used. These changes are done only by doing installation from ISO. Both changes are supposed to speed-up overall read/write operations, mainly for log servers and managements.

Then I would promote Secondary management to be Primary and repeat the same steps on former Primary (now Secondary).