One of the Firewall down in the cluster caused network blackout

[VarunTP]

We have a network outage since all our gateway is in firewall and its a cluster of 3. 

While reviewing the output of the command below, the second firewall was highlighted as "down." It was uncertain whether it was completely offline or in a hung state, as the indicator lights were blinking despite the "DOWN" status. Notably, even though the first and third firewalls were evenly distributing a 50% load each, all services were inaccessible. After rebooting the second firewall and reconnecting all cables from the core switch to the firewall, it successfully came back online. Following this, high availability (HA) and all network services were restored..

Any suggestions on this issue 

 SG-CHKPFW-6200-1-ch01-01> show cluster state

Cluster Mode:   HA Over LS

ID         Unique Address  Assigned Load   State          Name

1 (local)  x.x.2.1       33%             ACTIVE         SG-CHKPFW-1-ch01-01

2          x.x.2.2       33%             ACTIVE         SG-CHKPFW-1-ch01-02

3          x.x.2.3       33%             ACTIVE         SG-CHKPFW-1-ch01-03

During the issue :

SG-CHKPFW-6200-1-ch01-01> show cluster state

Cluster Mode:   HA Over LS

ID         Unique Address  Assigned Load   State          Name

1 (local)  x.x.2.1       50%             ACTIVE         SG-CHKPFW-6200-1-ch01-01

2          x.x.2.2         0%             DOWN         SG-CHKPFW-6200-1-ch01-02

3          x.x.2.3       50%             ACTIVE         SG-CHKPFW-6200-1-ch01-03

[PhoneBoy]

What version/JHF are the gateways?
What method of ClusterXL Load Sharing is being used and what was the CPU utilization during the outage?
You can use cpview or the “What’s The Story” view in Health Check Point as a starting point: https://support.checkpoint.com/results/sk/sk171436

[CheckPointerXL]

Try to verify if member 2 has been is Active Lost Lost state during outage.

It sounds like a split brain issue

[Chris_Atkinson]

Are two members of the cluster enough to handle the peak traffic load?

[the_rock]

You may need TAC case about this for further investigation.

Best regards,

Andy