Traffic drop after upgrading MHO1

[Prasaddere]

Hello,

We have single site dual orchestrator Scenario whereas MHA1 and MHO2 and 3 SGM.

We have upgraded the MHO1 and MHO2 to latest 81.20 with JHF#76 successfully

First we have upgraded MHO2 to 81.20 and then JHF#76, there was no issue reported. But when we upgraded MHO1 to latest 81.20, team reported that some of the application behind firewall stopped working for 20 or more minutes where MHO1 tooks only 10min for upgrade and reboot of the device. till we had not installed the hotfix #76. Later automatically this application started working. Then we installed the hotfix #76 on MHO1. If anyone have idea on this behavior. 

What happens to traffic if any MHO is getting rebooted or down? 

Does the traffic disconnect on failed MHO and connect back through the other MHO.? how much time it takes for the same?

[AkosBakos]

Hi @Prasaddere 

What were the exact steps of the MHO upgrade?

1. MHO1 to R81.20
2. MHO2 to R81.20
3. MHO1 take 76
4. MHO2 take 76

Or you merged the upgarade and JHF install step without failover?

Akos

[Prasaddere]

Step1: MHO2 Upgrade to 81.20 and Reboot

Step2: MHO2 Hotfix #76 and reboot

Step3: MHO1 Upgrade to 81.20 and reboot

Step4: MHO1 Hotfix 76 and reboot

[AkosBakos]

Hi @Prasaddere 

Did you do manual failver, or it happaned manually during the upgrade?

Akos

[Prasaddere]

we did not do any failover, it happened during upgrade.

[AkosBakos]

If you check the connections with this command : g_fw ctl conntab  you need to find the connection at least on 2 SGMs

echo the output into a file and compare it. Maybe this not belongs to the upgrade, and it is a independent issue on Maestro.

cphaprob syncstat doesn't show issue?

Akos

[Prasaddere]

Thanks Akos for your inputs,

Two queries,

1.what happens to traffic which is passing through MHO in case reboot/down, does is disconnect that MHO and does it start passing through Other MHO and in how much time?

2. During upgrade should out orchd stop and then we should upgrade then start orchd. What is the recommended.? 

[AkosBakos]

The two MHO must be substitute of each other. Maybe the switch which is the neigbour of the MHO would't have learn the "new" MAC-s? 

[AkosBakos]

That would be a good info, does itreproducible? If yes, tho TAC can help in the investigation further. Are the two MHO-s directly interconnected?
I would search around the sync, maybe this is sync issue what is still persists.

Akos

[Lari_Luoma]

Hello,

Both MHOs are always active and there is no failover mechanism between them. Do you have bond interfaces defined in a way that each bond has members from each MHO? While it is impossible to tell the exact reason for the outage without examining the logs, I believe your outage is most likely related to the bonds.