SSH host key changed, unable to login to upgraded R81.20 MDS

[cosmos]

I had hoped I never needed to come back here again, but I'm mid-upgrade and the Check Point has hit the proverbial fan (make of that what you will).

After battling snapshot and lvm limitations, I managed to clear out all the unnecessary things blowing out my snapshots including core dumps, random backups in home folders and an enormous amount of temporary files left over from who knows what (3-4 years old). Cleared out the unused global policy IPS profile junk, took enough mds backups (more core dumps found and mds backups created thanks to the ever so eager cpinfo...)

Ran the preupgrade verifier (which I might add instructions for which must be found in the R81 documentation, it's disappeared from the R81.20 doco along with an incorrect statement where to reassign global policies), all looks good.

Exported all the backups (including snapshot) to another MDS, imported Check_Point_R81.20_T634_Fresh_Install_and_Upgrade.tar, and proceeded with the install. Unlike cpinfo which asks you if you want to include core dumps (and does anyway even if you say no), the upgrade starts immediately. No warning that clients will disconnect, the server will reboot and you'll need the new console but we've been doing this for 30 years so we know the drill.

On connecting to the server post reboot, what do we have here? A new SSH host key? Odd but ok, let's accept it and login. Only SSH login fails, "access denied". Let's try the VM console.... same deal - it's not client ACLs or ciphers, it's authentication. Not even remote auth, these are all local users.

Try reboot into maintenance mode... Grub asks for credentails, no go. Try restore snapshot, Grub still asking for credentails.

I've logged it with TAC but I don't see them helping. Fortunately we took a VM snapshot before all this palaver, it may be our saving grace.

But why? You could ask Why Check Point, that's another story. Let me dig up my BOFH excuse generator for Check Point...

Where are you Bob, The Rock, Vikas, Phoneboy!

[cosmos]

Here's why.... installer install upgrade xx

[exits community with tail between legs]

[Lesley]

You clean installed it right? That you have to use default username / password

Check_Point_R81.20_T634_Fresh_Install_and_Upgrade. -> upgrade for inplace upgrade the first for clean install 😉 

[cosmos]

Yeah that was not the intention... while we should all be paying attention, I feel the process is counterintuitive...

Upgrade DA:       >installer agent install xx
Install HFA:      >installer install xx
Upgrade OS:       >installer upgrade xx
Clean install OS: >installer install

I also noticed an upgrade warns you that the system will be upgraded, configuration is preserved, requires a reboot and gives you the option to opt out. A clean install just starts - no warnings that you just selected clean install if you really did want to upgrade.

Changing this behaviour would have a positive impact on weary engineers 🙂

[Bob_Zimmerman]

Incidentally, when the underlying OS gets an update, it trashes the existing SSH keys and generates new ones. I don't remember if R81 to R81.20 was one of these, but R81.20 to R82 is, R82 to R82.10 is, and I'm pretty sure R82.10 to R82.20 will be. Super annoying. And if you watch the console the first time it boots the new OS, it actually generates new keys seven or eight times, which is separately weird. Maybe it's following outdated advice to let the system build up enough entropy in the software PRNG, even though every processor Check Point has shipped in over a decade has a good hardware RNG.

[cosmos]

😆

These were VMs, from R80.40 to R81.20 upgrade they retained keys. Appliances tomorrow (VSX)...