topic Re: One of the Firewall down in the cluster caused network blackout in Firewall and Security Management

One of the Firewall down in the cluster caused network blackout

VarunTP — Thu, 23 Nov 2023 10:25:01 GMT

We have a network outage since all our gateway is in firewall and its a cluster of 3.

While reviewing the output of the command below, the second firewall was highlighted as "down." It was uncertain whether it was completely offline or in a hung state, as the indicator lights were blinking despite the "DOWN" status. Notably, even though the first and third firewalls were evenly distributing a 50% load each, all services were inaccessible. After rebooting the second firewall and reconnecting all cables from the core switch to the firewall, it successfully came back online. Following this, high availability (HA) and all network services were restored..

Any suggestions on this issue

SG-CHKPFW-6200-1-ch01-01> show cluster state

Cluster Mode: HA Over LS

ID Unique Address Assigned Load State Name

1 (local) x.x.2.1 33% ACTIVE SG-CHKPFW-1-ch01-01

2 x.x.2.2 33% ACTIVE SG-CHKPFW-1-ch01-02

3 x.x.2.3 33% ACTIVE SG-CHKPFW-1-ch01-03

During the issue :

SG-CHKPFW-6200-1-ch01-01> show cluster state

Cluster Mode: HA Over LS

ID Unique Address Assigned Load State Name

1 (local) x.x.2.1 50% ACTIVE SG-CHKPFW-6200-1-ch01-01

2 x.x.2.2 0% DOWN SG-CHKPFW-6200-1-ch01-02

3 x.x.2.3 50% ACTIVE SG-CHKPFW-6200-1-ch01-03

Re: One of the Firewall down in the cluster caused network blackout

PhoneBoy — Thu, 23 Nov 2023 15:42:41 GMT

What version/JHF are the gateways?
What method of ClusterXL Load Sharing is being used and what was the CPU utilization during the outage?
You can use cpview or the “What’s The Story” view in Health Check Point as a starting point: https://support.checkpoint.com/results/sk/sk171436

Re: One of the Firewall down in the cluster caused network blackout

CheckPointerXL — Fri, 01 Dec 2023 23:35:03 GMT

Try to verify if member 2 has been is Active Lost Lost state during outage.

It sounds like a split brain issue

Re: One of the Firewall down in the cluster caused network blackout

Chris_Atkinson — Sat, 02 Dec 2023 02:17:52 GMT

Are two members of the cluster enough to handle the peak traffic load?

Re: One of the Firewall down in the cluster caused network blackout

the_rock — Sat, 02 Dec 2023 16:57:34 GMT

You may need TAC case about this for further investigation.

Best regards,

Andy