Cluster broke down - SMB 9000

[Exonix]

Hello all,

it's me again 🙂

Now we have a problem with another cluster of two SMB 9000. It was working good, then something happened. Maybe I did something wrong. Now the only one Node is online, another one doesn't respond, although I see its MAC. I even can't open the web-interfaces of the broken Node.

Some logs from the nodes. The working node doesn't see the neighbor:

[fws02:0]# cphaprob roles
ID         Role
2 (local)  Master

[fws02:0]# cphaprob state
Cluster Mode:   High Availability (Active Up) with IGMP Membership
ID         Unique Address  Assigned Load   State          Name                               
2 (local)  172.27.255.150  100%            ACTIVE         fws02

Active PNOTEs: LPRB
Last member state change event:
   Event Code:                 CLUS-114904
   State change:               ACTIVE(!) -> ACTIVE
   Reason for state change:    Reason for ACTIVE! alert has been resolved
   Event time:                 Sat Apr 12 11:17:48 2025

Cluster failover count:
   Failover counter:           0
   Time of counter reset:      Sat Apr 12 10:17:24 2025 (reboot)

 The broken node:

[fws01:0]# cphaprob roles
ID         Role
1 (local)  Non-Master
2          Master

[fws01:0]# cphaprob state
Cluster Mode:   High Availability (Active Up) with IGMP Membership
ID         Unique Address  Assigned Load   State          Name
1 (local)  172.27.255.149  0%              DOWN           fws01
2          none            100%            ACTIVE         fws02

Active PNOTEs: FSYNC, POLICY, IAC
Last member state change event:
   Event Code:                 CLUS-112400
   State change:               INIT -> DOWN
   Reason for state change:    FULLSYNC PNOTE - Policy installation failure
   Event time:                 Sat Apr 12 11:19:12 2025

Cluster failover count:
   Failover counter:           0
   Time of counter reset:      Sat Apr 12 11:17:27 2025 (reboot)

Very appreciate your help 🙏

[Exonix]

I don't understand this... it started working it self...

[Chris_Atkinson]

Did you install policy recently?

Please share the firmware version & build of the gateways in question.

[the_rock]

@Exonix Chris made an excellent point there...even on regular Gaia cluster, clustering will NEVER work until you install the policy.

Andy

[the_rock]

So its complaining about the policy failure and sync...what does cphaprob symcstat show, as well as fw stat?

Andy