VSX Incorrect configuration - Local cluster member...

lomi_drevo

Good day folks,

I have a problem with VSX cluster running R81.10 - 7000 appliance Cluster Mode: VSX High Availability (Active Up) with IGMP Membership

secondary node reports problem with interfaces after reboot

***

[Expert@fwtest2:0]# cphaprob stat

Cluster Mode: VSX High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 10.253.13.1 100% ACTIVE fwtest1
2 (local) 10.253.13.2 0% DOWN fwtest2

Active PNOTEs: IAC

Last member state change event:
Event Code: CLUS-110800
State change: INIT -> DOWN
Reason for state change: Incorrect configuration - Local cluster member has fewer cluster interfaces configured compared to other cluster member(s)
Event time: Wed Feb 19 15:10:23 2025

***

node2 reports 11 required interfaces

[Expert@fwtest2:0]# cphaprob -a if

vsid 0:
------
CCP mode: Manual (Unicast)
Required interfaces: 11
Required secured interfaces: 1

Interface Name: Status:

Mgmt UP
bond0 (S-LS) UP

S - sync, HA/LS - bond type, LM - link monitor, P - probing

Virtual cluster interfaces: 1

Mgmt 10.0.3.115

node1 reports 13 required interfaces

[Expert@fwtest1:0]# cphaprob -a if

vsid 0:
------
CCP mode: Manual (Unicast)
Required interfaces: 13
Required secured interfaces: 1

Interface Name: Status:

Mgmt UP
bond0 (S-LS) UP

S - sync, HA/LS - bond type, LM - link monitor, P - probing

Virtual cluster interfaces: 1

Mgmt 10.0.3.115

BUT, both nodes report having the same number of UP interfaces:

[Expert@fwtest1:0]# cphaprob -vs all -a if |grep UP
Mgmt UP
bond0 (S-LS) UP
wrpj320 UP
wrpj448 UP
wrpj128 UP
wrpj192 UP
wrpj256 UP
wrpj384 UP
bond0 (S-LS) UP
bond3.306 (LS) UP
bond0 (S-LS) UP
bond2.364 (LS) UP
bond0 (S-LS) UP
bond3.643 (LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond2.565 (LS) UP
bond0 (S-LS) UP

[Expert@fwtest2:0]# cphaprob -vs all -a if |grep UP
Mgmt UP
bond0 (S-LS) UP
wrpj320 UP
wrpj448 UP
wrpj128 UP
wrpj192 UP
wrpj256 UP
wrpj384 UP
bond0 (S-LS) UP
bond3.306 (LS) UP
bond0 (S-LS) UP
bond2.364 (LS) UP
bond0 (S-LS) UP
bond3.643 (LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond2.565 (LS) UP
bond0 (S-LS) UP

vsx stat -v looks fine on both nodes:

[Expert@fwtest1:0]# vsx stat -v
VSX Gateway Status
==================
Name: fwtest1
Access Control Policy: fwtest_VSX
Installed at: 19Feb2025 14:59:06
Threat Prevention Policy: fwtest_VSX
SIC Status: Trust

Number of Virtual Systems allowed by license: 10
Virtual Systems [active / configured]: 6 / 6
Virtual Routers and Switches [active / configured]: 1 / 1
Total connections [current / limit]: 42395 / 6279300

Virtual Devices Status
======================

Type: S - Virtual System, B - Virtual System in Bridge mode,
R - Virtual Router, W - Virtual Switch.

[Expert@fwtest2:0]# vsx stat -v
VSX Gateway Status
==================
Name: fwtest2
Access Control Policy: fwtest_VSX
Installed at: 19Feb2025 15:09:25
Threat Prevention Policy: fwtest_VSX
SIC Status: Trust

Number of Virtual Systems allowed by license: 50
Virtual Systems [active / configured]: 6 / 6
Virtual Routers and Switches [active / configured]: 1 / 1
Total connections [current / limit]: 41516 / 6279300

Virtual Devices Status
======================

Type: S - Virtual System, B - Virtual System in Bridge mode,
R - Virtual Router, W - Virtual Switch.

******

I couldn't find any hint of what is wrong so I assumed there might be something wrong with node2 itself so I tried fresh install of node2 and subsequent vsx_reconfigure. After a lot of unexpected struggles I ended up with exactly the same issue after fresh install. Only thing I haven't tried is reboot of node1, but som VSs are used as production(despite it's name) so I'm a bit hesitant to do it.

I have ran out of ideas of what to try or where to look. Any hint would be greatly appreciated

lomi_drevo

Managed to fix not so clear problem with even less clear solution. Out of desperation I renewed certificates on cluster object and 2 VS(ipsec blade is not enabled there so it should be just cosmetic issue). "cphaprob stat" showed vsx issue instead of interface. "cphaprob -l list" pointed to problematic vsid 1 (virtual router). Installed policy on vsid 1 and voila from DOWN-> Standby. Beats me.

Chris_Atkinson

Which JHF is the cluster installed with, was the reboot part of an upgrade activity?

Comparing your outputs above VS1 has no policy...

CCSM R77/R80/ELITE

lomi_drevo

A bit older one is in place: HOTFIX_R81_10_JUMBO_HF_MAIN Take: 139. It wasn't part of an upgrade activity. I was just playing with fwaccel dos on internet facing VS(just external interface) and wanted to make sure I implemented it right so it survives reboot.

Are you a member of CheckMates?

VSX Incorrect configuration - Local cluster member has fewer cluster interfaces configured