topic Re: speed up VSX upgrade R81.10 => R81.20 in Firewall and Security Management

speed up VSX upgrade R81.10 => R81.20

Wolfgang — Tue, 13 Aug 2024 13:24:11 GMT

Recently we upgraded a VSX cluster from R81.10 to R81.20. Everything was fine, no problems, no connections broken. Main problem was the time it took to upgrade the whole cluster. Some log reading und investigation shows that the upgrade procedure was running about 20 minutes for every virtual system, regardless if this was only a virtual switch, - router or a virtual firewall.

We are on the way to upgrade another system with about 20 virtual systems. Any ideas to speed up this upgrade process ? The expected timeline will expand our maintenance schedule.

The upgrade was done on a pair of 15600 appliances with no CPU utilization problems.

Re: speed up VSX upgrade R81.10 => R81.20

Bob_Zimmerman — Tue, 13 Aug 2024 13:36:24 GMT

Which part took the time? The vsx_util upgrade on the management, or actually upgrading the members?

If it was the vsx_util upgrade, you can kick that off several hours ahead of your upgrade window. It's just updating the management database and rebuilding all of the policies.

If the part which took all the time was actually upgrading the members, what upgrade method did you use?

Re: speed up VSX upgrade R81.10 => R81.20

Wolfgang — Tue, 13 Aug 2024 13:46:24 GMT

@Bob_Zimmerman upgrading the members was the problematic long running part. We did a "Multi-Version Cluster (MVC) upgrade"

Re: speed up VSX upgrade R81.10 => R81.20

Bob_Zimmerman — Tue, 13 Aug 2024 13:54:30 GMT

In-place ('installer upgrade'), or did you do a clean install then reprovision? The latter is how I do most VSX upgrades (including some R81.10 to R81.20), and they're consistently pretty fast.

Re: speed up VSX upgrade R81.10 => R81.20

Bob_Zimmerman — Tue, 13 Aug 2024 13:58:24 GMT

Just had a thought. The 15600 shipped with spinning rust by default. Most of my VSX systems have SSDs. I would expect dealing with each VS to involve a lot of small-file operations, so that could make a pretty big difference in upgrade duration.

Re: speed up VSX upgrade R81.10 => R81.20

Wolfgang — Tue, 13 Aug 2024 14:01:36 GMT

We did an inplace upgrade. I think "clean install then reprovision" will be much faster. But our next system to upgrade will be a Maestro/VSX environment and I don't know this will be possible?

Re: speed up VSX upgrade R81.10 => R81.20

Alex- — Tue, 13 Aug 2024 20:49:26 GMT

We tend to do in-place upgrade of VSX R81.10 to R81.20 with Blink and MVC and as you mention it works well.

Our preferred methods of upgrading VSX is normally format and reconfigure. However with R81.20 management, we encounter quite frequently the issue that it fails at the end with "Operation ended with errors" and the workaround is to run "reset_gw" on the member that was formatted and restart the vsx_util procedure. This always works on the second run.

So we gave a shot to Blink and it went perfectly fine.

With large systems, running this twice takes a lot of time even on modern appliances.

Re: speed up VSX upgrade R81.10 => R81.20

emmap — Wed, 14 Aug 2024 02:05:00 GMT

The other thing thing about using the Blink files for in-place upgrades is that there are a number of boot time fixes in the JHF that will be included, plus you don't then have to do a separate JHF install afterwards.

Unfortunately there's not a lot we can do about slow HDDs if that's what's in the boxes but at least using Blink is something.

Re: speed up VSX upgrade R81.10 => R81.20

Wolfgang — Wed, 14 Aug 2024 07:28:58 GMT

We used the blink image with included JHF, as a result no extra reboots are needed. The timesteps of the logfiles shows the time needed for upgrading a VS (see attached screenshot). There was no high disk utilization during the upgrade process, but maybe this is the limitation.