Fetching policy failed when running sp_upgrade script

[Martijn]

Hi all,

This week we tried to upgrade a Maestro set from R81.10 take 165 to R81.20 take 84. Because the Security Groups are configured with LACP bonds, I could not use the Zero Downtime (MVC) procedure and used the Minimum Downtime procedure. 

I followed the steps described in the R81.20 Maestro Admin guide and all seems to go OK. Orchestrators (MHO-140) where upgraded without any issues. Also the upgrade of the first member in the Security Group went OK and was succesfully upgraded to R81.20. But then things went wrong.

When running the sp_upgrade script on the upgraded member, the fetching of the policy failed:

Fetching the policy from the Management Server and installing it... Failed on members 1_1

Fetching the policy from xx.xx.xx.xx and installing it... Failed on members 1_1
Enter the IP address of the Management Server that manages this Security Group:xx.xx.xx.xx

Fetching the policy from xx.xx.xx.xx and installing it... Failed on members 1_1

Try to run 'sp_upgrade' again in few seconds. If the problem persists, contact support.

I have tried to run the command with the --NO-MVC option and got the same problem.

Performed a verify and that seems to be OK.

[Expert@xxxxxxxxxxx-ch01-01:0]# sp_upgrade --verify
Starting the Security Group Pre-Upgrade Verifier:
Enter the IP address of the Management Server that manages this Security Group:xx.xx.xx.xx
Cluster State: Failed
Check the state of the following Security Group members. Make sure they are in Active state before proceeding with the upgrade or remove them from the Security Group.
1_01
Connectivity to Management Server: Passed
LACP: Passed
Disk Space: Passed
The Security Group failed the Pre-Upgrade Verifier tests. Do not continue with the upgrade until you fix all the detected issues.

The upgraded member was Down, but according to the Admin guide, that was expected. Also all verification earlier in the procedure reported an OK status.

In the end we had to revert the whole upgrade on the Security Group. Orchestrators are still on R81.20.

Did I miss something? Followed the procedure by the letter and double checked every step with the customer.
Anyone had the same issue when upgrading?

What can we do the next time we plan this upgrade? Are the LACP bonds affecting the upgrade?

Regards,
Martijn

[Lari_Luoma]

Do you manage the system over an LACP-bond? Try switching to magg or change the bonding type to active/backup temporarily.

[Martijn]

Hi,

I forgot to mention that. Production bonding groups (uplinks) are LACP. MAGG is Active/Backup.

Regards,
Martijn

[77Madness77]

Hi Martijn,

I have the same problem, SG is VSX and the connection from SG to MGMT is through a VS, I think the problem is that there is no connection to MGMT, so it can't take policies and when you upgrade it doesn't take policies, so it doesn't do anything.

How did you manage to solve it? I had this problem yesterday.

Regards

[emmap]

The connection between VSX and management server is through one of the VSs that the VSX hosts? This is not a supported deployment for any VSX setup, maestro or otherwise, upgrades will always fail in this scenario. 

[Martijn]

Hi,

No, VS0 is on the same network as the SmartCenter.

I think Implied Rules are the issue. Will try again with Implied Rules enabled.

Regards,
Martijn

[emmap]

I have heard of issues with VSX upgrades and implied rules being disabled, I think specifically the 'allow control connections' one.

[Machine_Head]

Just had a similar issue, solution was to manually copy the policy files from the Manager to the upgraded members and run sp_upgrade with fetch from tmp flag.

Solution was associated with sk180402.

I just think the upgraded members couldn't connect to the manager or viceversa for some reason.

My setup connects to the manager via uplink bond (acitve backup thoug), so maybe it doesn't like the fact that there is no management itnerface, not sure.