topic Re: Traffic drop after upgrading MHO1 in Hyperscale Firewall (Maestro)

Traffic drop after upgrading MHO1

Prasaddere — Sun, 22 Sep 2024 13:13:51 GMT

Hello,

We have single site dual orchestrator Scenario whereas MHA1 and MHO2 and 3 SGM.

We have upgraded the MHO1 and MHO2 to latest 81.20 with JHF#76 successfully

First we have upgraded MHO2 to 81.20 and then JHF#76, there was no issue reported. But when we upgraded MHO1 to latest 81.20, team reported that some of the application behind firewall stopped working for 20 or more minutes where MHO1 tooks only 10min for upgrade and reboot of the device. till we had not installed the hotfix #76. Later automatically this application started working. Then we installed the hotfix #76 on MHO1. If anyone have idea on this behavior.

What happens to traffic if any MHO is getting rebooted or down?

Does the traffic disconnect on failed MHO and connect back through the other MHO.? how much time it takes for the same?

Re: Traffic drop after upgrading MHO1

AkosBakos — Sun, 22 Sep 2024 15:50:48 GMT

Hi @Prasaddere

What were the exact steps of the MHO upgrade?

MHO1 to R81.20
MHO2 to R81.20
MHO1 take 76
MHO2 take 76

Or you merged the upgarade and JHF install step without failover?

Akos

Re: Traffic drop after upgrading MHO1

Prasaddere — Sun, 22 Sep 2024 15:57:38 GMT

Step1: MHO2 Upgrade to 81.20 and Reboot

Step2: MHO2 Hotfix #76 and reboot

Step3: MHO1 Upgrade to 81.20 and reboot

Step4: MHO1 Hotfix 76 and reboot

Re: Traffic drop after upgrading MHO1

AkosBakos — Sun, 22 Sep 2024 16:12:26 GMT

Hi @Prasaddere

Did you do manual failver, or it happaned manually during the upgrade?

Akos

Re: Traffic drop after upgrading MHO1

Prasaddere — Sun, 22 Sep 2024 16:17:33 GMT

we did not do any failover, it happened during upgrade.

Re: Traffic drop after upgrading MHO1

AkosBakos — Sun, 22 Sep 2024 16:26:36 GMT

If you check the connections with this command : g_fw ctl conntab you need to find the connection at least on 2 SGMs

echo the output into a file and compare it. Maybe this not belongs to the upgrade, and it is a independent issue on Maestro.

cphaprob syncstat doesn't show issue?

Akos

Re: Traffic drop after upgrading MHO1

Prasaddere — Sun, 22 Sep 2024 16:42:07 GMT

Thanks Akos for your inputs,

Two queries,

1.what happens to traffic which is passing through MHO in case reboot/down, does is disconnect that MHO and does it start passing through Other MHO and in how much time?

2. During upgrade should out orchd stop and then we should upgrade then start orchd. What is the recommended.?

Re: Traffic drop after upgrading MHO1

AkosBakos — Sun, 22 Sep 2024 16:54:24 GMT

The two MHO must be substitute of each other. Maybe the switch which is the neigbour of the MHO would't have learn the "new" MAC-s?

Re: Traffic drop after upgrading MHO1

AkosBakos — Sun, 22 Sep 2024 16:59:39 GMT

That would be a good info, does itreproducible? If yes, tho TAC can help in the investigation further. Are the two MHO-s directly interconnected?
I would search around the sync, maybe this is sync issue what is still persists.

Akos

Re: Traffic drop after upgrading MHO1

Lari_Luoma — Mon, 23 Sep 2024 18:10:18 GMT

Hello,

Both MHOs are always active and there is no failover mechanism between them. Do you have bond interfaces defined in a way that each bond has members from each MHO? While it is impossible to tell the exact reason for the outage without examining the logs, I believe your outage is most likely related to the bonds.