VSX Incorrect configuration - Local cluster member has fewer cluster interfaces configured

[lomi_drevo]

Good day folks,

I have a problem with VSX cluster running R81.10 - 7000 appliance Cluster Mode: VSX High Availability (Active Up) with IGMP Membership

secondary node reports problem with interfaces after reboot

***

[Expert@fwtest2:0]# cphaprob stat

Cluster Mode: VSX High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 10.253.13.1 100% ACTIVE fwtest1
2 (local) 10.253.13.2 0% DOWN fwtest2

Active PNOTEs: IAC

Last member state change event:
Event Code: CLUS-110800
State change: INIT -> DOWN
Reason for state change: Incorrect configuration - Local cluster member has fewer cluster interfaces configured compared to other cluster member(s)
Event time: Wed Feb 19 15:10:23 2025

***

node2 reports 11 required interfaces

[Expert@fwtest2:0]# cphaprob -a if

vsid 0:
------
CCP mode: Manual (Unicast)
Required interfaces: 11
Required secured interfaces: 1

Interface Name: Status:

Mgmt UP
bond0 (S-LS) UP

S - sync, HA/LS - bond type, LM - link monitor, P - probing

Virtual cluster interfaces: 1

Mgmt 10.0.3.115

node1 reports 13 required interfaces

[Expert@fwtest1:0]# cphaprob -a if

vsid 0:
------
CCP mode: Manual (Unicast)
Required interfaces: 13
Required secured interfaces: 1

Interface Name: Status:

Mgmt UP
bond0 (S-LS) UP

S - sync, HA/LS - bond type, LM - link monitor, P - probing

Virtual cluster interfaces: 1

Mgmt 10.0.3.115

BUT, both nodes report having the same number of UP interfaces:

[Expert@fwtest1:0]# cphaprob -vs all -a if |grep UP
Mgmt UP
bond0 (S-LS) UP
wrpj320 UP
wrpj448 UP
wrpj128 UP
wrpj192 UP
wrpj256 UP
wrpj384 UP
bond0 (S-LS) UP
bond3.306 (LS) UP
bond0 (S-LS) UP
bond2.364 (LS) UP
bond0 (S-LS) UP
bond3.643 (LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond2.565 (LS) UP
bond0 (S-LS) UP

[Expert@fwtest2:0]# cphaprob -vs all -a if |grep UP
Mgmt UP
bond0 (S-LS) UP
wrpj320 UP
wrpj448 UP
wrpj128 UP
wrpj192 UP
wrpj256 UP
wrpj384 UP
bond0 (S-LS) UP
bond3.306 (LS) UP
bond0 (S-LS) UP
bond2.364 (LS) UP
bond0 (S-LS) UP
bond3.643 (LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond0 (S-LS) UP
bond2.565 (LS) UP
bond0 (S-LS) UP

vsx stat -v looks fine on both nodes:

[Expert@fwtest1:0]# vsx stat -v
VSX Gateway Status
==================
Name: fwtest1
Access Control Policy: fwtest_VSX
Installed at: 19Feb2025 14:59:06
Threat Prevention Policy: fwtest_VSX
SIC Status: Trust

Number of Virtual Systems allowed by license: 10
Virtual Systems [active / configured]: 6 / 6
Virtual Routers and Switches [active / configured]: 1 / 1
Total connections [current / limit]: 42395 / 6279300

Virtual Devices Status
======================

ID | Type & Name | Access Control Policy | Installed at | Threat Prevention Policy | SIC Stat
-----+-------------------------+-----------------------+-----------------+--------------------------+---------
1 | R EFR_tst | EFR_tst_policy | 30Jul2024 14:35 | EFR_tst_policy | Trust
2 | S DVS_tst | DVS_tst_policy | 19Feb2025 15:55 | DVS_tst_policy | Trust
3 | S TVS_tst | TVS_tst_policy | 19Feb2025 14:22 | TVS_tst_policy | Trust
4 | S LVS_tst | LVS_tst_policy | 17Feb2025 13:18 | LVS_tst_policy | Trust
5 | S RAS | RAS_tst | 19Feb2025 15:23 | RAS_tst | Trust
6 | S IVS_tst | IVS_tst_policy | 19Feb2025 15:55 | IVS_tst_policy | Trust
7 | S RAS2 | RAS2_tst | 19Feb2025 15:22 | RAS2_tst | Trust

Type: S - Virtual System, B - Virtual System in Bridge mode,
R - Virtual Router, W - Virtual Switch.

[Expert@fwtest2:0]# vsx stat -v
VSX Gateway Status
==================
Name: fwtest2
Access Control Policy: fwtest_VSX
Installed at: 19Feb2025 15:09:25
Threat Prevention Policy: fwtest_VSX
SIC Status: Trust

Number of Virtual Systems allowed by license: 50
Virtual Systems [active / configured]: 6 / 6
Virtual Routers and Switches [active / configured]: 1 / 1
Total connections [current / limit]: 41516 / 6279300

Virtual Devices Status
======================

ID | Type & Name | Access Control Policy | Installed at | Threat Prevention Policy | SIC Stat
-----+-------------------------+-----------------------+-----------------+--------------------------+---------
1 | R EFR_tst | EFR_tst_policy | 19Feb2025 15:10 | <No Policy> | Trust
2 | S DVS_tst | DVS_tst_policy | 19Feb2025 15:55 | DVS_tst_policy | Trust
3 | S TVS_tst | TVS_tst_policy | 19Feb2025 15:10 | TVS_tst_policy | Trust
4 | S LVS_tst | LVS_tst_policy | 19Feb2025 15:10 | LVS_tst_policy | Trust
5 | S RAS | RAS_tst | 19Feb2025 15:23 | RAS_tst | Trust
6 | S IVS_tst | IVS_tst_policy | 19Feb2025 15:55 | IVS_tst_policy | Trust
7 | S RAS2 | RAS2_tst | 19Feb2025 15:22 | RAS2_tst | Trust

Type: S - Virtual System, B - Virtual System in Bridge mode,
R - Virtual Router, W - Virtual Switch.

******

I couldn't find any hint of what is wrong so I assumed there might be something wrong with node2 itself so I tried fresh install of node2 and subsequent vsx_reconfigure. After a lot of unexpected struggles I ended up with exactly the same issue after fresh install. Only thing I haven't tried is reboot of node1, but som VSs are used as production(despite it's name) so I'm a bit hesitant to do it.

I have ran out of ideas of what to try or where to look. Any hint would be greatly appreciated

[lomi_drevo]

Managed to fix not so clear problem with even less clear solution. Out of desperation I renewed certificates on cluster object and 2 VS(ipsec blade is not enabled there so it should be just cosmetic issue). "cphaprob stat" showed vsx issue instead of interface. "cphaprob -l list" pointed to problematic vsid 1 (virtual router). Installed policy on vsid 1 and voila from DOWN-> Standby. Beats me.

[Chris_Atkinson]

Which JHF is the cluster installed with, was the reboot part of an upgrade activity?

Comparing your outputs above VS1 has no policy...

[lomi_drevo]

A bit older one is in place: HOTFIX_R81_10_JUMBO_HF_MAIN Take: 139. It wasn't part of an upgrade activity. I was just playing with fwaccel dos on internet facing VS(just external interface) and wanted to make sure I implemented it right so it survives reboot.