Cluster Sync lost after Get Interfaces with topology

[Maarten_Sjouw]

Last week we ran into an issue that a cluster was loosing the sync status and the backup member went into down state.

Investigation learned that the Get Topology command erased the sync status of the Sync interfaces. There is no verification on the SmartConsole for this anymore, in R77.30 there was no way you could get out of the Topology window without a Sync interface set.

In R80.10 you can and you can push policy without any notification.

[Kaspars_Zibarts]

ouch! did you manage to find out why did it remove it from topology? was it missing on the gateway? was it set up manually before or with the get? just curious.

we have some other weird examples of R80.10 not doing good verification and pushing out changes that shouldn't be allowed (object creation with the same name..) but not related to topology. will write once I had more time to sum it up

[Maarten_Sjouw]

It was probably setup manually before, one of my colleagues added a new interface and did the get topology and then did not check all interfaces properly, when we tried to get the added interface running in a change window, we found that the cluster was having issues and the interface VIP did not get activated, not pingable nor showing in cphaprob -a if Then after a while we found that the sync interface was no longer set as a cluster 1st Sync. Restoring theat and pushing policy activated the VIP on the additonal interface but we are still having some issues with the cluster itself. We ran out the change window and they shut the switch ports again which left the cluster in problem state. even removing the added interface in the SmartConsole did not resolve this yet, waiting for the next window.

[PhoneBoy]

If you can reproduce this, I recommend opening a TAC case.

My understanding is that we're still using SmartDashboard-style code for editing gateway/cluster objects due to the fact they still operate in CPMI.

Thus I would expect they would have all the same checks in place.

[Maarten_Sjouw]

Oh yeah very simply reproducible, just got to any cluster and change the Sync interface to private and click Ok, push policy and only then I did see one verification error: 
Regardless of state synchronization, Cluster FWC must have a trusted (secured) network defined for normal clustering operations.

[JozkoMrkvicka]

Lession learned:

Do NOT click on "get interfaces from topology" in any case.

Did it once and spent around 4 hours to check all 800 VLANs to find missing IP...

Of course I didnt do database revision because... all will be fine

Lession learned volume II:

Do database revision (no valid for R80)...

[PhoneBoy]

The "do database revision" also does not apply with VSX (which never supported database revisions).

[Petr_Hantak]

I fully agree with your lessons because I had similar "hard way" learning couple years ago as well.

[Vladimir · Accepted solution]

Can we get a repeated prompts if the "Get Interfaces with Topology" is selected?

Something like:

1. Are you sure you want to get Interfaces with Topology? It's been known to cause some unexpected surprises.

and

2. Have you freed-up your evening or weekend or have decided to change your occupation and the country of residence?

and

3. Have you at least made a snapshots and backups of the infrastructure you are working on?

[Maarten_Sjouw]

LOL, and what about this one:

1. Could you please check yourself if that you have at least 1 Sync interface?