unable to delete VS firewalls when VSX members are offline

[Jacques_Spelier]

Hi there,

We seem to have created an issue and wondering if anybody has come across similar situation and how they resolved it.

We have decomm'ed our VSX appliances before deleting the VS/VSX objects in SmartCenter. We are now stuck in a loop where the SC is trying to communicate to the VSX members to inform them of the deletion. We opened a ticket with TAC and they directed us to delete the reference of the VS via GUIDBEDIT.  We have remove the vs_slot_members references as per SK127232. When we reattempt to delete the VS in question, still generates communication attempts and fails to remove the object. Also since we did the DBedit change as recommended by TAC, we can no longer push policy to other non-vsx gateways. Message to the effect of "A Cluster (VSNAME) cannot be empty. It must have cluster members."

I am thinking that a deletion of the "vs_slots_objects" container located under the "Network Objects"  could resolve our issue but looking for feedback from the community. We are waiting on TAC right now.

Thoughts/suggestions?

Thanks.

[Bob_Zimmerman]

Deleting things in GuiDBEdit may have left you in an inconsistent state which may require more GuiDBEdit work to fix. Export your config right now to be sure you can always get back to this point at worst.

If this sort of thing comes up in the future, you should run these commands on your management server:

fw debug fwm on TDERROR_ALL_VSXM_DBG_SKIP_PING=INFO
fw debug fwm on TDERROR_ALL_VSXM_DBG_SKIP_INSTALL=INFO
fw debug fwm on TDERROR_ALL_VSXM_DBG_SKIP_PULL_SIC=INFO

Together, they let you make management-side changes without needing communications with the VSX boxes. Good for deleting VSs when you can't reach the cluster.

[Jacques_Spelier]

Hi Bob,

Did as you suggested with the debug cmds. Some of the warning messages disappeared by at the end still got an error message and the VS did not get deleted.

"Checking connection with VSX

Deleting VSNAME

Internal error: Number of members should be greater then 1

Virtual System Deletion Completed with Errors.

We tried to delete a "non-modified" VS object on that same VSX cluster with same results.

[Bob_Zimmerman]

Like I mentioned, I think those commands will probably only help in the future. I think the GuiDBEdit changes have broken your database enough to require manual intervention to fix it. If you can restore back to a management export, snapshot, or whatever taken before you made changes with GuiDBEdit, the debug flags might help.

[Jacques_Spelier]

Hi Bob,

Just an update.

We restored to a snapshot and got back to a known good state.  enabled the debug and was able to move through and delete all the VSX environment successfully.

Thanks for the help.  All good now.

P.S. This is why I love Checkmates.

[Bob_Zimmerman]

Nice! Good to hear you had a backup from before the dbedit changes.

I've had to use these debugs in the past when upgrading a cluster member. The VSX object on the SmartCenter had incorrect interface definitions. For example, the SmartCenter listed eth4 and eth5 as interfaces, but they were members of bond1, which was also listed. When running 'vsx_util reconfigure', they were no longer seen as valid interfaces. I only had so many physical ports on the box, so I couldn't just rearrange some. I had to remove the missing interfaces from the object, but when I tried, it failed because it couldn't connect to the member I had upgraded. Dumb problem. Fortunately, the fix is pretty simple.