80.40->81.20 Minimum Downtime Upgrade via Smart Console

[VikingsFan]

Fairly new to Check Point and was doing my first upgrade of a Open Server cluster running R80.40.  Was following this doc: https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Installation_and_Upgrade_Gui... and it mentions best practice was using the central deployment in SmartConsole.  Great I thought, seems straightforward.  My boss had never used the Smart Console method and always did the step by step procedure but was willing to give the new method a try.

I went into Smart Console, right clicked the R80.40 cluster and chose a version upgrade to R81.20 Take 53.  Everything checked out fine, it kicked off the standby node and started upgrading the server.  Server install finished, it reboots and comes back online running R81.20.  Everything on the standby appears healthy but the task details in Smart Console is just stuck at 100% for the standby and saying "updating products."  We let it sit there a good 20-30 minutes and nothing moved.

Decided to try and bounce the standby one more time and as soon as it started rebooting the Smart Console task changed to 'rebooting' for the standby node and then appeared to try and resume the upgrade steps but because the standby node was down/rebooting they failed.  The task failed out and we ended up just going the 'tried and true' method of the manual steps but I was wondering if we just had a unique situation or is the Smart Console upgrade not ready for prime time yet?

I do have another two clusters to upgrade and will most likely be choosing the Web GUI method and not Smart Console this time. 😄

Thanks!

[the_rock]

I did same method in lab once and never again. I found, and this is just me personally, though it seems all is moving along smoothly, process fails at the end and its not as user friendly as I envisioned.

I always now stick with web UI method, as it never let me down 🙂

Best,

Andy

[VikingsFan]

Appreciate the feedback.  I'm hard headed and had to find out on my own. 🙂    Luckily, the standby was upgraded fine and we just resumed the upgrade steps through the web with no downtime.

Everyone advised me that they only had ever used the web ui upgrade method and it was solid but I saw in the document the big bolded Best Practice so wanted to give it a whirl.  Lesson learned and I'll be doing web UI upgrades from now on. lol

[the_rock]

The way I look at it is this...no matter what, of course change is always good and welcome, BUT, personally, if I know certain method works well, just my personal opinion, why change it? : - )

Andy

[PhoneBoy]

The upgrade process via SmartConsole has improved substantially since R80.40.

[VikingsFan]

My single management server was on 81.20 and the two FWs on 80.40.  Are you saying now that we're all on 81.20 the process will be improved for future upgrades or doing the upgrade on a 81.20 management but on 80.40 servers would have been improved also?

Sorry for the confusion but wasn't sure which one you meant.  Thanks!

[the_rock]

Phoneboy will clarify for you, but Im sure he meant process has improved since R80.40, meaning in R81+...just my logical thinking.

Andy

[VikingsFan]

Thanks.  I assumed that's what he meant... now that I'm on R81.20, moving forward things should be a lot better.  Maybe I'll give it one last chance next time. 😄

[the_rock]

Thats why even-ng and Azure labs are the best, super easy to build and replicate things. I will let you know how things went with jumbo 53 install, just doing it now 🙂

Andy

[the_rock]

Current progress, lets see if it goes well...fingers crossed

Andy

 

[the_rock]

@VikingsFan 

Standby is done, now its upgrading active. 35 mins in, active shows at 16%, so its moving along 🙂

Andy

[VikingsFan]

Hah, sweet.  Don't think you mentioned but which version are you coming from?

[the_rock]

Its R81.20 cluster...it was on jumbo 43, updating to jumbo 53

Andy

[the_rock]

Just finished, all good! Took 1 h and 52 seconds 🙂

Andy

[VikingsFan]

Very nice!  I may have been a little too ambitious doing a 80.40 jump to 81.20... probably feel more comfortable doing jumbo/hotfix patches with Smart Console.  Thanks for trying it out.

[the_rock]

I would not be comfortable myself using CDT for say big upgrades like that, but say R81 to R81.10 or R81.10 to R81.20 is okay.

No worries, glad to help mate, any time 🙂

Andy

[PhoneBoy]

I was thinking if you were upgrading the management from R80.40, not what you did.
My bad 🙂

Having said that, I know there is active work to improve this in R82.

[the_rock]

Im just installing jumbo 53 for R81.20 in Azure lab cluster using CDT, lets see how it goes, will report back : - )

Andy

[Boaz_Orshav]

Very sorry to hear this was your experience.

I'd appreciate if you can send me the some logs offline so we can understand what went wrong as I know many customer used this method for version upgrade

Will appreciate if you can run "collect_logs.bash" on the management machine and send me the tgz it creates to boazo@checkpoint.com

[VikingsFan]

Thanks Boaz!  I'll email you right now.  I'd prefer a secure upload location than email so will get my email address over to you.

[the_rock]

Let us know if anything good comes out of those logs, I would be curious to know.

Best,

Andy

[VikingsFan · Accepted solution]

Final update.  Thanks to Boaz for reviewing the logs and this is what he responded with:

  Took some time but the bottom line (there are some internal logs below if you wish to dig into it but not necessary):

1. Installation finished
2. Email sending started but got stuck for at least 15 minutes
3. Restart caused the installation status to change to “interrupted although the package was installed.

It’s kind of an issue as to when do we mark the installation as finished. In general the email sending is not part of the installation but it’s part of the “installation flow” so it caused an inconsistency between the package status (installed) and the installation status (interrupted)

We need to think of a better way to handle it but from your side the question is why the email sending got stuck (see below highlighted lines)

[the_rock]

Interesting...

[VikingsFan]

Hi Boaz,

I shot you an email yesterday.  Let me know if it gets to you.