Migration MDS R77.30 to R80.30 failed

[Maarten_Sjouw]

This weekend we had planned to migrate our 102 domain 3 server MDS setup, the primary went just fine, ran for 8 hours with 31 domains. Next I installed jumbo on the 2 secondary's (had nothing else to do) and when I tried to migrate a secondary server it completely failed, so you think due to the jumbo difference, so also installed jumbo on the primary and attempted the import again, still fails and fails and fails. (as soon as it failed I nvolved TAC)

That was yesterday, today we got some special attention from Israel and we had a fun day trying to get the secondary to migrate without any result.

In the end they found a cpm crashdump, back to R&D with that, first question was: did they install a jumbo? 

Lesson to learn DO NOT in any case during migration to R80.30 install a Jumbo!!!

[Ryan_Ryan]

I can confirm this (well fairly similar), experienced this first hand aswell with R77.30 to R80.20.

The MLM would not complete its upgrade (after 3 hours of being stuck in the upgrade in process screen at 58% it would revert itself back to R77.30), it was due to us having upgraded the MDM, and putting the latest jumbo on it, All we did was remove the jumbo from the manager, then the MLM upgraded just fine. Then we jumbo'd the manager, then jumbo'd the log server and all was fine (just many hours wasted!)

I forget which elg file now, but somewhere hidden in the thousands of lines of messages was something about versions mismatching.

[Maarten_Sjouw]

Ryan,
In your case it was a mismatch, in my case I matched the jumbo versions, the message regarding R80.30, for now do not install any jumbo when you are migrating until the migration is done!

[Eran_Habad]

Hi @Maarten_Sjouw ,

My name is Eran and I'm a Group Manager in the R&D of Check Point, responsible for the core I/S of the Management Server including the Management upgrade process.

First, I apologize for the bad experience you had with the upgrade to R80.30. Me and my team are working constantly to improve the quality and robustness of the Management upgrade process. We're reviewing thoroughly cases of failures, like the one you experienced, to learn from them and to make sure that you (and rest of our customers) will enjoy a smooth upgrade process. This is my responsibility. 

I also want to say upfront, that when performing an advanced upgrade, we highly recommend to install the latest recommended JHF before starting the import process. We have high confidence in the JHF, and we see it as our most important platform for delivering quality fixes, including fixes for the upgrade process. You're also invited to read my post with few tips and recommendations regarding the Management JHF. As you wrote, it's important that when installing JHF it will be installed on all machines before starting the upgrade process.

Indeed in the JHF of R80.30, we just found an issue with one of our fixes. The symptom includes MDS upgrade failure from R7X to R80.30, when the upgrade is done as advanced (export/import and not CPUSE) with JHF (take 19) installed. 

Such degradation is extremely rare, and we're working fast to provide a resolution. A new ongoing take of JHF will be released shortly with a fix for this issue (will be published in sk153152). I will also update this post when it's out. Meanwhile, MDS customers with R7X can upgrade to R80.30 via CPUSE or to upgrade using export/import but without JHF installed.

I would like again to express my regret for the failure you had. R80.30 is a great version with many important features and enhancements, and I'm sure you will enjoy it. Don't hesitate to contact me for any matter (here or privately).

Eran.

EDIT: The issue has been fixed in JHF Take #50. See: https://supportcenter.us.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&so...

[Maarten_Sjouw]

Eran,

I know R80.30 is a good version to go to, I already have another MDS running with around 40 domains on it.
I'm happy that you say it should be kinda mandatory to install the JHF before starting the migration. Unfortunately we ran into the problem that was there and had to postpone our migration by 2 weeks.
We will do the migration on plain vanilla R80.30 machines and run the JFH after the migration is done.
Thank and regards.

[Yifat_Chen]

Hi all, 

New Jumbo was released today with a fix for this issue (Take #50) 

 https://community.checkpoint.com/t5/Product-Announcements/R80-30-JHF-New-Ongoing-Take-50/ba-p/61759 

[Maarten_Sjouw]

Last weekend we did a new attempt to this migration and we were successful in the total migration.
There were some small issues we ran into but with a direct line to TAC with R&D in the background we were able to solve them all in time for production on Monday.

Many thanks to the teams at Check Point for assisting us during this big job. Special thanks to escalation engineer Asaf for putting up with me.

One thing for future MDS migration candidates: keep in mind that the migration will take a long time, in our case the primary MDS migration took 8 hours, 1 for the MDS database and 7 for all 31 domains. The 2 secondary MDSs took 7,5 hour each, including around 35 domains each.
1 other thing to keep in the back of your head: when using user.def to setup per VPN topologies, the user.def is copied to the conf dir, however it does not take into acount that your gateways are not upgraded yet and does not copy it to the compatibility file for R77.30
Also the pre-upgrade verifier does not see the changed file and does not notify you as it does for implied.def or crypt.def.

All together it went quite well.