SMB cluster XL detailed logs

[vladdar]

Hello all,

I am having a hard time to find detailed logs on cluster XL events on a pair of Quantum Spark 1800 with R80.20.40 (992002691). I know the firmware is a bit older but I faced issues when trying to upgrade to R81.10 (basically I was unable to form a cluster after an upgrade).

There is a state change in output bellow and I can't find a reason what happened and what it means State change: ACTIVE(!) -> ACTIVE

How can I find more details on this platform ? There is nothing interesting in /var/log/messages and 

Gaia clish <show cluster member> doesn't work on Spark.

Thank You.

[Expert@gw1]#cphaprob stat

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State

1 (local) 10.231.149.1 100% ACTIVE
2 10.231.149.2 0% STANDBY

Active PNOTEs: None

Last member state change event:
Event Code: CLUS-114904
State change: ACTIVE(!) -> ACTIVE
Reason for state change: Reason for ACTIVE! alert has been resolved
Event time: Fri Nov 3 13:09:43 2023

Last cluster failover event:
Transition to new ACTIVE: Member 2 -> Member 1
Reason: Incorrect configuration - Local cluster member has fewer cluster interfaces configured compared to other cluster member(s)
Event time: Fri Nov 3 10:59:24 2023

Cluster failover count:
Failover counter: 20
Time of counter reset: Fri Nov 3 13:09:43 2023 (reboot)

[Expert@gw1]# cphaprob show_failover

Last cluster failover event:
Transition to new ACTIVE: Member 2 -> Member 1
Reason: Incorrect configuration - Local cluster member has fewer cluster interfaces configured compared to other cluster member(s)
Event time: Fri Nov 3 10:59:24 2023

Cluster failover count:
Failover counter: 20
Time of counter reset: Fri Nov 3 13:09:43 2023 (reboot)

Cluster failover history (last 20 failovers since reboot/reset on Fri Nov 3 13:09:43 2023):

No. Time: Transition: CPU: Reason:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Fri Nov 3 10:59:24 2023 Member 2 -> Member 1 00 Incorrect configuration - Local cluster member has fewer cluster interfaces configured compared to other cluster member(s)
2 Thu Oct 5 20:54:38 2023 Member 1 -> Member 2 00 Incorrect configuration - Local cluster member has fewer cluster interfaces configured compared to other cluster member(s)
3 Sun Sep 17 12:49:25 2023 Member 2 -> Member 1 00 Interface LANBOND0 is down (Cluster Control Protocol packets are not received)
4 Sun Sep 17 11:44:53 2023 Member 1 -> Member 2 00 Incorrect configuration - Local cluster member has fewer cluster interfaces configured compared to other cluster member(s)

[Chris_Atkinson]

Which process did you follow in attempting the upgrade?

You can try checking the following log(s) for additional information to start, but you may need to consult further with TAC.

/var/log/messages*

$FWDIR/log/sfwd.elg*

[vladdar]

Thank you, looks like I have found something in $FWDIR/log/sfwd.elg*

nothing in /var/log/messages*

Altough, I do not fully understand it. Something happened in that vlan, could it affect the whole cluster (other vlans) ? Unfortunately, logs on primary member have rotated so I don't have them.

[sfwd 9105 4154539136]@gw2[3 Nov 13:03:33] Cluster member status changed from STANDBY to DOWN.
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Op=route-del
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Dst=10.0.0.96
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Mask=255.255.255.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Op=route-del
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Dst=10.0.0.104
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Mask=255.255.255.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Op=route-del
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Dst=10.0.0.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Mask=255.255.255.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Op=route-del
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Dst=10.0.6.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Mask=255.255.255.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Op=route-del
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Dst=10.2.0.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Mask=255.255.0.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:34] Cluster member status changed from DOWN to STANDBY.
[sfwd 9105 4154539136]@gw2[3 Nov 13:03:37] Requesting settings synchronization from cluster peer.

[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Op=route-add
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Dst=10.0.0.96
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Mask=255.255.255.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Op=route-add
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Dst=10.0.0.104
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Mask=255.255.255.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Op=route-add
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Dst=10.0.0.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Mask=255.255.255.248
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Op=route-add
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Dst=10.0.6.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Mask=255.255.255.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Gateway=10.3.3.2
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Detected route change:
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Op=route-add
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Interface=LANBOND0.3
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Dst=10.2.0.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Mask=255.255.0.0
[sfwd 9105 4154539136]@gw2[3 Nov 13:09:44] Gateway=10.3.3.2

[PhoneBoy]

Any changes to the interfaces/VLANs is likely to impact cluster state.