topic Re: Upgrading form R81.10 to R81.20 in General Topics

Upgrading form R81.10 to R81.20

Dave_F — Tue, 04 Apr 2023 19:12:35 GMT

I am upgrading my firewalls appliances members in a cluster from R81.10 T87 to R81.20.

I am using the Installation and Upgrade Guide R81.20 page 417 on Zero Downtime Upgrade of a Security Gateway Cluster and it is not accurate!!

Procedure:
1. On each Cluster Member, change the CCP mode to Broadcast
Important - This step does not apply to R80.30 with Linux kernel 3.10 (run the
"uname -r" command).
Best Practice - To avoid possible problems with switches around the cluster
during the upgrade, we recommend to change the Cluster Control Protocol
(CCP) (CCP) mode to Broadcast.
Step Instructions
1 Connect to the command line on each Cluster Member.
2 Log in to the Expert mode.
3 Change the CCP mode to Broadcast:
[Expert@FW1:0]# cphaconf set_ccp broadcast
Notes:
This change does not require a reboot.
This change applies immediately and survives reboot.

Actual output:

1) First issue

[Expert@FW1:0]# cphaconf set_ccp broadcast

CCP mode is only unicast
This command is obsolete

[Expert@FW1:0]#

which is ok if that is not problem since it is only unicast now.

2) do we need to remove the standby member from the cluster and add it back in?

If so what is cli commands to add/remove member from a cluster?

runnning "clusterxl_admin down

Actual output:

[Expert@FW1:0]# clusterxl_admin down
bash: clusterxl_admin: command not found
[Expert@FW1:0]#

4 Make sure the CCP mode is set to Broadcast:
cphaprob -a if

2. On the Cluster Member M2, upgrade to R81.20 with CPUSE, or perform a Clean Install of
R81.20
Important - You must reboot the Cluster Member after the upgrade or clean install.

With all that being said above what is the "exact procedure" since it is not listed?

Regards

Dave F

Re: Upgrading form R81.10 to R81.20

Noa_Moe — Tue, 04 Apr 2023 22:45:47 GMT

The last upgrade we did from 80.40 to 81.10 these commands were accurate. I'm also curious on the new process for Zero Downtime upgrades.

Re: Upgrading form R81.10 to R81.20

genisis__ — Tue, 04 Apr 2023 21:38:55 GMT

I would use "set cluster member mvc on" on the R81.20 device, then once both members are running R81.20 turn it off.

Re: Upgrading form R81.10 to R81.20

the_rock — Tue, 04 Apr 2023 22:42:58 GMT

I was once on a call with guy from R&D about something unrelated and I told him that line about changing CCP mode should either be removed or fixed, he said they would do it, its been 6 years now and its still there. I cant even count how many cluster upgrades I did and I never changed ccp mode at all and everything worked fine.

Andy

Re: Upgrading form R81.10 to R81.20

Dave_F — Tue, 04 Apr 2023 22:44:49 GMT

Thanks for the reply I do have a question both nodes are at R81.10 T87 so neither are at R81.20 unless "set cluster member mvc on" works on R81.10? I am only worried about the upgraded device taking control while the Active node is still R81.10.

Thanks, Dave F

Re: Upgrading form R81.10 to R81.20

the_rock — Tue, 04 Apr 2023 22:51:50 GMT

I just recently did cluster upgrade from R81.10 to R81.20 exact same way I had done it for years, never changed ccp mode, mvc, nothing. Upgraded current backup, rebooted, changed version in smart console for cluster to R81.20, unchecked "dont push to cluster...", after policy install, failed over from current master to upgraded member by running clusterXL_admin down on R81.10 member, upgraded that fw, rebooted, rechecked the option I mentioned when pushing policy, made sure all was good, failed back to original master member.

Thats it, thats all, plain and simple, no issues : - )

I have perfectly working cluster in the lab on R81.10 jumbo 94, happy to do any testing for you, no issues. Currently those VMs are off as my colleague needed more resources for Aruba training, but I can always power them on after work hours and test, happy to do it.

Re: Upgrading form R81.10 to R81.20

Dave_F — Tue, 04 Apr 2023 22:49:08 GMT

Thanks for the reply what I am concerned about after upgrading the standby node first (R81.10 -> R81.2) and it becomes the active node while Active node running R81.10 Also thinks it is the active node.

Regards, Dave F

Re: Upgrading form R81.10 to R81.20

the_rock — Tue, 04 Apr 2023 22:58:49 GMT

OK, not to exaggerate now, but I had done this probably 50 times and every time I did it, below is what I followed and NEVER had an issue (forget about the versions, process was literally the same even 20 years ago lol)

I will list steps I do:

-generate backups on both cluster members (I dont bother with snapshots, but thats your choice)

-push policy before the upgrade, just to make sure all is good

-download say R81.20 (latest recommended jumbo) and verify it can be upgraded

-install R81.20 on BACKUP member and reboot

-once rebooted, change cluster version in dashboard to R81.20 and uncheck that option for cluster "dont install policy..." and once policy is done, it will ONLY apply on upgraded member

-current master is STILL master, one on R81.10

-issue failover by running clusterXL_admin down on R81.10 member, so everything goes to upgraded member and verify this by running cphaprob roles and cphaprob state commands

-if all good, upgrade R81.10 member to R81.20, install policy after reboot, just recheck option "dont install policy..." (its also indicated what option that is in zero downtime upgrade)

-once done, verify cluster state and if all good, you can fail back to original master member (just dont forget to run clusterXL_admin up) : - )

-once done, verify all...routing, nat, vpn etc

I am 100% POSITIVE in these steps, because they never failed for me.

Andy

Re: Upgrading form R81.10 to R81.20

Bob_Zimmerman — Wed, 05 Apr 2023 00:03:40 GMT

If you have a separate management server, just upload the R81.20 CPUSE package into its repository and upgrade using SmartConsole. It handles all of the steps for you. Seriously, it's great. Saves so much headache.

For your question about removing a member from the cluster and adding it back, no, you absolutely don't remove a member from the cluster.

For your question about forcing a failover, it's clusterXL_admin. The XL is uppercase.

Be aware that a "zero-downtime upgrade" is probably not what you think it is. It means there is no point at which new connections cannot be formed, but it drops ongoing connections. You probably want a multi-version cluster upgrade, which involves the 'cphaconf mvc on'.

Re: Upgrading form R81.10 to R81.20

the_rock — Wed, 05 Apr 2023 03:36:35 GMT

Personally, never found it any faster doing the process from smart console.

Re: Upgrading form R81.10 to R81.20

Wolfgang — Wed, 05 Apr 2023 05:03:55 GMT

Like @Bob_Zimmerman said, I would prefer using Smartconsole, that's the best and easiest way to do all the steps of an upgrade for you. One thing to mentioned, before you do the upgrade you have to check the cluster recovery settings, this must be set to "Maintain current active Cluster Member". There are some connectivity effects during the upgrade with the other setting if you start the process with the wrong member. But with "Maintain current active Cluster Member" you are save.

Re: Upgrading form R81.10 to R81.20

the_rock — Wed, 05 Apr 2023 12:55:45 GMT

Good point about that screenshot @Wolfgang . Its just me, but personally, I prefer to stick with method that I know works 100%, which is upgrade from web UI. Not saying smart console method is worse, I know it works, but I just dont find it any faster or more useful, but again, thats just me.

Cheers,

Andy

Re: Upgrading form R81.10 to R81.20

Dave_F — Wed, 05 Apr 2023 13:57:54 GMT

Thanks Wofgang I did double check my ClusterXL settings and it is set as you have mentioned.

Regards, Dave F

Re: Upgrading form R81.10 to R81.20

Dave_F — Wed, 05 Apr 2023 14:05:40 GMT

Thanks Rock well I guess I do not have your luck as recently I was upgrading from R80.40 T(most current) to R81.10 and I had 2 active nodes one with R80.40 and one with R81.10 fighting to be the one. The not so quick fix was to power down, unplug the R81.10 FW, blow it it away and reinstall as a new RMA'd device replacement.

Regards, Dave F

Re: Upgrading form R81.10 to R81.20

Dave_F — Wed, 05 Apr 2023 14:08:10 GMT

Thanks Bob_Zimmerman noted I will add to my memory banks as I do forget to do commands exactly as noted.

Regards,

Dave F

Re: Upgrading form R81.10 to R81.20

the_rock — Wed, 05 Apr 2023 14:08:52 GMT

Not sure what to tell you Dave, sorry...every time I did this, whenever current backup was upgraded and rebooted, current master was ALWAYS still master, which makes perfect sense, even though it had lower software version, since rebooting backup member with higher version would never make it become master.

Andy

Re: Upgrading form R81.10 to R81.20

Dave_F — Wed, 05 Apr 2023 14:09:19 GMT

Agreed 🙂 Things are always slower when you are watching 😄

Re: Upgrading form R81.10 to R81.20

the_rock — Wed, 05 Apr 2023 14:10:45 GMT

I heard that LOL

Re: Upgrading form R81.10 to R81.20

the_rock — Wed, 05 Apr 2023 14:22:23 GMT

By the way, this is option I forgot to attach screenshot of I as referring to in my steps.

Andy

Re: Upgrading form R81.10 to R81.20

Bob_Zimmerman — Wed, 05 Apr 2023 14:47:17 GMT

For me, it's faster both in terms of wall-clock time for one upgrade (doesn't rely on a person watching stuff to trigger the next step) and in terms of how many upgrades we can do in a month. Only a handful of people on my team have ever installed Gaia from scratch. If every upgrade requires somebody who knows what 'cphaconf mvc on' does, then one of the leads needs to be personally involved in a lot of upgrades, making them the limiting factor.