Firmware update warning with R81.20 upgrade on Azure

[nmelay2]

Hi all,

I need to update a customer's Azure-hosted management and log servers from R81.10 to R81.20.
Following R81.20 documentation, I went the in-place upgrade path, as per sk177714.
(For some reason, the regular CPUSE upgrade packages do not show up on Azure VMs, and you need to use specific packages from this SK).
Everything looked fine but then Verify Update gave me a very confusing message:

Based on a system check, a firmware update may be necessary on one or more network cards to bring them up to date with the current Gaia network drivers. This update is a one-time process which could take several minutes, and is executed after Gaia OS upgrade is finished and has rebooted post-upgrade. After the firmware update is complete, the system will automatically reboot once more to apply the new firmware. Please do not reboot or shut the system down during this time.

The need for a firmware update on an Azure VM is rather unexpected.
However, I learned Azure VM do indeed run on hardware with Mellanox NICs, as do CP appliances, and part of the real NIC is actually exposed to the VM.
https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-how-it-works

Did you guys run into this?
Can we fearlessly go on with the upgrade in this situation?
I wouldn't want the upgrade process to fail trying to flash a new NIC firmware, and either just crash here or enter an endless reboot loop...

And before anyone asks, yes I'm also running this through TAC.
I'm just looking for insight from fellow admins with hands-on experience with this use case.

[Don_Paterson]

I haven't seen it, but am curious to know if you considered deploying a new R81.20 SMS (followed by an import)?

You would get the latest R81.20 marketplace image/template.

It is just out of curiosity that I ask, with no knowledge of the specific deployment or requirements, including logging, which may be one reason why you want to do the in-place upgrade.

Regards,

Don

[nmelay2]

That's what we did for previous upgrades, but going through all of this seemed unnecessarily painfull.

Deploying a new SMS does come with the benefit of the new properly aligned disk layout though, so I'm still considering it.

[Don_Paterson]

ACK
Too soon for R82 😉

[the_rock]

Way too soon 🙂

[Don_Paterson]

This time is different 😉

But seriously, with an EA that lasted nearly a year, it could be something to consider.

Obviously carefully considering all the factors!

[nmelay2]

I'm not sure why you're bringing R82 here.

AFAICT, Check Point has been really good with the quality of GA releases since R80.40, but I still wouldn't adopt it so soon without a very strong customer requirement to do so.

[the_rock]

It probably wont be recommended until summer 2025...just my educated guess.

Andy

[JozkoMrkvicka]

My wild guess is that very soon we will get first JHF for R82 and until CPX there will be JHF Take around 30 which will be consided as "stable" and thus R82 will be recommended during/after CPX 😉

R81.20 was released 21.11.2022 and marked as recommended 27.7.2023.

[the_rock]

Lets see 😉

Andy

[Alex-]

We got the notification our Smart-1 Cloud SMS is planned for upgrade in December. 

We will then be able to test management with R82 Take 0 if no JHF comes in and gets Recommended in such a short timeframe, let's hope it will sail smoothly. 😀

[Don_Paterson]

Just in case it helps:

I see that someone else had this situation last year, but on physical appliances.

https://community.checkpoint.com/t5/Security-Gateways/Enterprise-appliance-upgrade-to-R81-20/m-p/190...

[nmelay2]

Yeah, on real hardware, this warning actually makes sense.
But on a VM running on hardware owned by Microsoft, there's no way Check Point will be able to flash its own firmware.
So it's going to tail, either smoothly or badly.

The lack of feedback here and from TAC seems to indicate not that many customers actually do in-place upgrades on Azure VMs... while it seems to be the currently recommended upgrade path.
I indeed did not know until recently this was even an option.

[the_rock]

Keep in mind that up until one point in 2023, it was only possible to do in place upgrade on mgmt server in Azure, I believe, not gateways. Now, its possible on both, but I do agree with you, documentation about this could be better.

This is an official sk about it.

Andy

https://support.checkpoint.com/results/sk/sk177714

[nmelay2]

Andy...

This thread is specifically about upgrading mgmt and logs servers, and I mentionned sk177714 in the first two lines of my first post.
So, yeah, I'm keeping that in mind.

[the_rock]

Fair enough. I used that same process few times and never had an issue. What did TAC say?

[nmelay2]

Nothing so far, they're still asking for unrelated logs instead of answering my very simple "is this a known issue?" question.
But the case has only been running for 5 days, lol.
I'll keep you updated next month. 😉

[the_rock]

For what its worth, I will answer it myself, haha. No, I dont believe its a known issue, at least from my experience, but it would help to get an official statement from the vendor.

Yes, keep us posted mate 🙂

Andy

[nmelay2 · Accepted solution]

OK, quick update.

2 weeks after I opened the case (and escaladed it), I finally got an answer from TAC that "there shouldn't be an issue with the upgrade" and we should just go ahead.

So today we went for it, and the R81.20 upgrade package installed just fine.
After the installation, I searched logs for an attempted firmware upgrade, and indeed found the two following lines in /var/log/messages:

Dec 4 10:01:21 2024 vmcprdckp0001 mlx_logger: Starting Mellanox firmware verification / updater tool.
Dec 4 10:01:21 2024 vmcprdckp0001 mlx_logger: Cloud-based platform detected. Therefore, no Mellanox cards will be updated.

So finally, this configuration (Azure VM) is actually properly and explicitly handled by the firmware upgrade process.
It's really a shame the upgrade package pre-install verification incorrectly warns about this!

But there's more.
After the R81.20 upgrade was finalized, I went for the recommended JHF install.
But JHF 89 would not install. Installation would not even start, and CPUSE froze on "initiating install" for at least half an hour, with absolutely no sign of activity.
After reboot, a new Verify Update (which succeeded initially) now failed with the following message:

R81.20 Jumbo Hotfix Accumulator Recommended Jumbo Take 89
Installation is not allowed.
Reason:
The previous installation of this package terminated abnormally. To install this package, contact Check Point Support for assistance.
Contact Check Point Technical Services for further assistance.

I looked at the last not-yet-recommended JHF release notes, and saw no mention of this issue.
Tried to contact TAC, then after a few hours with no response, I still tried installing JHF 90, because hell, why not.
Amazingly, this actually worked, and JHF 90 installed just fine.

All in all, the whole R81.20 mgmt upgrade on Azure was really not as a pleasant experience as it should have been.
On the good side, my customer was happy that I handled this for him. 😉

[Don_Paterson]

It's about 10 years since vSec/CloudGuard became available in Azure and upgrading has only been supported in the last couple.

Destroy and redeploy was repeated over and over...

It gets interesting when you compare management pricing for on-prem vs public cloud vs MaaS Smart-1 Cloud.

Apart from Regulations (restrictions) and log retention requirements it is easy to see the attraction to cloud (and maybe Smart-1 Cloud specifically, for management).

With no Regulatory requirements and budget for extra log retention Smart-1 Cloud looks attractive.

Painful to say that because I am old school and like physical on-prem.

"The Cloud" makes things more difficult way too often.

[the_rock]

I guess like anything in life, we have to adapt 🙂

Andy

[Don_Paterson]

I also wonder about the number of SMS deployments in public cloud.

The 'Check Point Reference Architecture for Azure' SK does not actually state that the PAYG license is an SM25.

It does not seem to be documented anywhere.

I put some feedback in for the SK.

 https://support.checkpoint.com/results/sk/sk109360 

It seems like it would be documented if there were more enquiries and they needed to clarify in the SK.

Otherwise, we have to assume that there is a lot of BYOL or few deployments of SMS in Azure.

[nmelay2]

This customer moved its infrastructure from datacenters to Azure (for better or for worse).
I assume many went the same path, and indeed brought in their own license.

[the_rock]

Every customer I know that did this went with BYOL approach.

[Don_Paterson]

Did you see Smart-1 Cloud adoption based on cost analysis?

Meaning that on-prem and BYOL was abandoned because of Smart-1 Cloud cost and MaaS benefits (no upgrade burdens and Support included in the price).

The Cloud First approach is common and now Smart-1 Cloud is recommended for CloudGuard SGs deployed in CSPs.

Obviously that is a general recommendation and customer owned management is still valid.

[the_rock]

Personally, I always recommend S1C approach these days, because if for one important thing, if there is need for emergency change, anyone who has access can do it from anywhere in the world. But, you are 100% correct, customer owned management is still valid, but most things are shifting towards cloud-based approach and Im totally on board with that.

Andy

[Don_Paterson]

The illiterate of the 21st century will not be those who cannot read and write but those who cannot learn, unlearn and relearn. 

[the_rock]

Ah...anyone can learn most things, as long as they put genuine effort into it. Just my honest opinion.

Andy

[nmelay2]

For this customer, Smart-1 Cloud had too many limitations by then (authentication, IA, too slow, ...)

[the_rock]

I find its gotten way way better since 2020, when it was fairly new.

Andy