HA Management upgrade issues R80.40 --> R81.10

[David_C1]

Yesterday I attempted to upgrade our lab HA Smart-1 appliances from R80.40 to R81.10. I decided to do an in-place upgrade using CPUSE. High level procedure:
1. Upgrade CPUSE to latest on each appliance
2. Install (manually) the R81.10 upgrade tools package on each appliance
3. Run the upgrade verifier on each appliance. This did not report any issues on either appliance except to remind me to install JHFA after the upgrade
4. Take appropriate backups
5. Upgrade the primary/active. This completed without issue.
6. Install JHFA Take 87 on the primary
7. Confirm SmartConsole access, successfully pushed policy

This is where it got interesting.
8. Attempted to upgrade the standby, but no upgrade option was available. After verification was run again, I got this result:

(The primary server was upgraded and running).

I fought with this for a while. I even rolled back to a snapshot on the standby I created before starting any upgrades, but still was not given the option to upgrade the standby, only a clean install. I eventually went through with the clean install and everything is back up and running, but I curious if anyone else has seen this, I did something wrong, or I missed something in the upgrade documentation.

Thanks,

Dave

[Tal_Paz-Fridman]

Hi David

I have sent the details to relevant R&D owners so that they can look at the issue.

[David_C1]

I did find that ICMP was being dropped from the secondary mgmt server to the primary management server (the two SMS servers are in different datacenters behind firewalls, i.e. no unfiltered communication) during the general times I was attempting to upgrade the secondary. Could it be this simple?

Dave

[David_C1]

To answer my own question...yes, it can be that simple. After seeing the ICMP drops in the logs, I found sk179794, this tipped me off that the failed ICMP could be the cause. Apparently, as part of the secondary's verification that the primary has been upgraded, it tries to ping the primary. If this fails, the verification fails. Although it may be unusual to have HA management servers with firewalls between them, having this requirement in the upgrade documentation would have saved me several hours of work.

Dave

[Tal_Paz-Fridman]

Thank you for the update. There are definitely two issues we need to improve:

1) Improve message given in CPUSE stating that the Primary Management Server should be up and running but also reachable by the Secondary Management Server.

2) Add this requirement to the Installation and Upgrade Administration Guide.

I'll send the requirements to the relevant R&D owners.

[JozkoMrkvicka]

My suggestion is to be more specific what does it mean "Primary management is up and running and is reachable from Secondary management". Up and running means I see login prompt. But it doesnt say that all the proccesses must be up. The same for "reachable". What does it mean exactly ? I can ping them each other = is reachable? Or any API call must be successful ? Or ssh connection possible ? or some specific port must be listening ?

[_Val_]

Processes required for MGMT functionality are mentioned in sk97638, Management Server section. Ports required for management communication are listed in sk52421, management section again. 

[David_C1]

Neither sk mentions ICMP. At least sk52421 should be updated to reflect this requirement for management HA environments.

Dave

[JozkoMrkvicka]

I faced exactly the same issue. The issue was that I have installed Jumbo Hotfix on upgraded Primary Management while Secondary was not yet upgraded. You need to upgrade Primary while NOT installing any hotfix. Then upgrade Secondary. Once Secondary is upgraded, install all needed hotfixes on Primary, then on Secondary.

It is also mentioned in the upgrade guide:

Important - Before you can install Hotfixes on servers that work in Management High Availability, you must upgrade all these servers.

[David_C1]

Interesting...after I allowed ICMP between the managers, I was able to upgrade the secondary (I rolled back to an R80.40 snapshot on the secondary) and I had installed JHFA Take 87 on the primary before upgrading the secondary. No issues.

I will however likely adjust my procedure when it's time to upgrade the production managers.

Dave