topic Cluster inconsistency after policy installation in Firewall and Security Management

Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 11:41:17 GMT

Hello guys

We have noticed that after upgrading from R80.30 to R81.10 two of our clusters behave erratically after policy installation. The primary will remain as active(F) which believing the secondary is in standby while the secondary shows itself as active and the primary as Lost. This will go on for 4-5 minutes until both members converge in the correct state. cphaprob says that there are no ccp sent in the sync interface however tcpdump says otherwise

I noticed that the output of the following parameters is different on both members

[Expert@GW-01:0]# fw ctl get int fwha_mac_magic

fwha_mac_magic = 254

[Expert@GW-01:0]# fw ctl get int fwha_mac_forward_magic

fwha_mac_forward_magic = 253

[Expert@GW-02:0]# fw ctl get int fwha_mac_magic

fwha_mac_magic = 1

[Expert@GW-02:0]# fw ctl get int fwha_mac_forward_magic

fwha_mac_forward_magic = 254

Are this still relevant in R81.10, I have a case open with tac however it has been lagging so any opinion would be helpful

Re: Cluster inconsistency after policy installation

G_W_Albrecht — Thu, 14 Jul 2022 11:50:49 GMT

Your commands are outdated! Better look into the current CP_R81.10_ClusterXL_AdminGuide and here: sk42096: Cluster member is stuck in 'Ready' state

Re: Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 12:44:06 GMT

I did not see any mention of cluster ID or mac magic in the new admin guide. I am afraid that this is something the gateways have inherited from previous versions since they have been upgraded in place. On other R81.10 cluster I have done a clean install the values are the same on both members.

Re: Cluster inconsistency after policy installation

Chris_Atkinson — Thu, 14 Jul 2022 13:40:35 GMT

What CCP mode does each member believe it is operating in?

Note the following was introduced starting from R80.40

* Support for Cluster Control Protocol (CCP) in Unicast mode for any number of cluster members eliminating the need for CCP Broadcast, Multicast or Automatic modes.

* Eliminated the need for MAC Magic configuration when several clusters are connected to the same subnet.

* Cluster Control Protocol encryption is now enabled by default.

Re: Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 13:46:54 GMT

You're onto something while the active member has ccp correctly set as unicast on the standby it is like this

CCP mode: Manual (Multicast)

Re: Cluster inconsistency after policy installation

Chris_Atkinson — Thu, 14 Jul 2022 13:50:35 GMT

They should ideally both be the same i.e. both auto or both unicast.

Re: Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 13:50:46 GMT

Any idea as to how I can change it?

Re: Cluster inconsistency after policy installation

the_rock — Thu, 14 Jul 2022 13:51:03 GMT

I checked in client's cluster and both members have exact same values, mac_magic as 254 and mac_forward_magic as 253.

Can you paste output of cphaprob state and cphaprob -a if?

Re: Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 13:52:20 GMT

gw2

cphaprob -a if

CCP mode: Manual (Multicast)
Required interfaces: 5
Required secured interfaces: 1

Interface Name: Status:

eth5 UP
eth8 UP
Sync (S) UP
bond0.11 (LS) UP
bond0.48 (LS) UP

S - sync, LM - link monitor, HA/LS - bond type

Virtual cluster interfaces: 16

gw1

cphaprob -a if

CCP mode: Manual (Unicast)
Required interfaces: 5
Required secured interfaces: 1

Interface Name: Status:

eth5 UP
eth8 UP
Sync (S) UP
bond0.11 (LS) UP
bond0.48 (LS) UP

S - sync, LM - link monitor, HA/LS - bond type

Virtual cluster interfaces: 16

Re: Cluster inconsistency after policy installation

the_rock — Thu, 14 Jul 2022 13:56:01 GMT

[Expert@HostName]# cphaconf set_ccp unicast

I dont believe multicast is supported in R81.10 from what I remember last time customer and I tried changing it, but give it a go. Its been few months, so its possible it was another protocol.

Re: Cluster inconsistency after policy installation

Chris_Atkinson — Thu, 14 Jul 2022 14:02:33 GMT

To set the CCP mode:

In Gaia Clish, run:
set cluster ccp {auto | unicast | multicast | broadcast}
In Expert mode, run:
cphaconf set_ccp {auto | unicast | multicast | broadcast}

This configuration applies immediately and survives reboot.

Re: Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 14:06:10 GMT

I change it however the issue still happens this is what cphaprob state on gw2 looks after policy installation. There is no unique address for gw-1 either

ID Unique Address Assigned Load State Name

1 none 0% GW-01
2 (local) a.b.c.d 100% GW-02

Active PNOTEs: LPRB, IAC

Last member state change event:
Event Code: CLUS-116505
State change: DOWN -> ACTIVE(!)
Reason for state change: All other machines are dead (timeout), Interface Sync is down (Cluster Control Protocol packets are not received)
Event time: Thu Jul 14 16:59:52 2022

Last cluster failover event:
Transition to new ACTIVE: Member 1 -> Member 2
Reason: Available on member 1
Event time: Thu Jul 14 16:59:52 2022

Cluster failover count:
Failover counter: 6
Time of counter reset: Fri Jun 24 08:32:48 2022 (reboot)

Re: Cluster inconsistency after policy installation

the_rock — Thu, 14 Jul 2022 14:08:12 GMT

What mode did you change it to?

Re: Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 14:08:55 GMT

I changed both members to unicast

Re: Cluster inconsistency after policy installation

the_rock — Thu, 14 Jul 2022 14:09:40 GMT

So what is currently output of cphaprob state and cphaprob -a if?

Re: Cluster inconsistency after policy installation

MasterChief117 — Thu, 14 Jul 2022 14:10:31 GMT

CCP mode: Manual (Unicast)
Required interfaces: 5
Required secured interfaces: 1

same for both

Re: Cluster inconsistency after policy installation

the_rock — Thu, 14 Jul 2022 14:14:44 GMT

Okay and what about magic_mac settings? Are they the same? If so, then its possible to make it work, you may need to do a quick failover by running clusterXL_admin down on master or cphastop and cphastart on BOTH.

Andy

Re: Cluster inconsistency after policy installation

charlokt — Thu, 14 Jul 2022 15:50:38 GMT

Same thing happened to me. I upgraded from R80.30 to R81.10 and the cluster was not established as it kept poking both members. Would it be necessary to configure the CCP mode so that the cluster remains Active > Passive? and keep it in the state it comes from R80.30?

Re: Cluster inconsistency after policy installation

charlokt — Thu, 14 Jul 2022 15:52:56 GMT

Re: Cluster inconsistency after policy installation

the_rock — Thu, 14 Jul 2022 16:35:07 GMT

I upgraded one customer's cluster from R80.40 to R81.10 and never had this problem. The mode has always been unicast.