topic Re: Reason for Firewall Failover in Firewall and Security Management

Reason for Firewall Failover

ESpataro — Thu, 15 Sep 2022 16:15:16 GMT

Can someone please explain what the FWD PNOTE means for failover and why it was triggered

Details below

[Expert@pccfw1:0]# cphaprob state

Cluster Mode: High Availability (Active Up) with IGMP Membership

ID Unique Address Assigned Load State Name

1 (local) 169.254.22.1 0% STANDBY pccfw1

2 169.254.22.2 100% ACTIVE pccfw2

Active PNOTEs: None

Last member state change event:

Event Code: CLUS-114802

State change: DOWN -> STANDBY

Reason for state change: There is already an ACTIVE member in the cluster (member 2)

Event time: Tue Sep 13 10:35:56 2022

Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2

Reason: FWD PNOTE

Event time: Tue Sep 13 10:35:26 2022

Cluster failover count:

Failover counter: 45

Time of counter reset: Wed Aug 18 14:04:42 2021 (reboot)

[Expert@pccfw1:0]# cphaprob show_failover

Last cluster failover event:

Transition to new ACTIVE: Member 1 -> Member 2

Reason: FWD PNOTE

Event time: Tue Sep 13 10:35:26 2022

Cluster failover count:

Failover counter: 45

Time of counter reset: Wed Aug 18 14:04:42 2021 (reboot)

Cluster failover history (last 20 failovers since reboot/reset on Wed Aug 18 14:04:42 2021):

No. Time: Transition: CPU: Reason:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 Tue Sep 13 10:35:26 2022 Member 1 -> Member 2 22 FWD PNOTE

2 Fri Sep 2 15:08:50 2022 Member 2 -> Member 1 13 ADMIN_DOWN PNOTE

3 Tue Aug 30 19:54:20 2022 Member 1 -> Member 2 05 FWD PNOTE

4 Sat Jun 18 11:10:51 2022 Member 2 -> Member 1 05 ADMIN_DOWN PNOTE

5 Fri Jun 17 22:12:24 2022 Member 1 -> Member 2 04 Interface eth3.29 is down (disconnected / link down)

6 Wed May 25 09:17:41 2022 Member 2 -> Member 1 30 ADMIN_DOWN PNOTE

7 Wed May 25 09:05:01 2022 Member 1 -> Member 2 18 Interface eth1-03 is down (Cluster Control Protocol packets are not received)

8 Tue May 24 23:49:22 2022 Member 2 -> Member 1 04 ADMIN_DOWN PNOTE

9 Tue May 24 23:40:38 2022 Member 2 -> Member 1 04 Member state has been changed after returning from ACTIVE/ACTIVE scenario (remote cluster member 1 has higher priority)

10 Tue May 24 23:30:05 2022 Member 2 -> Member 1 03 Member state has been changed after returning from ACTIVE/ACTIVE scenario (remote cluster member 1 has higher priority)

11 Tue May 24 23:30:04 2022 Member 1 -> Member 2 26 Interface is down (disconnected / link down)

12 Tue May 24 23:30:00 2022 Member 2 -> Member 1 24 Interface eth5 is down (disconnected / link down)

13 Tue May 24 23:29:47 2022 Member 1 -> Member 2 04 Interface is down (disconnected / link down)

14 Mon May 9 16:34:47 2022 Member 1 -> Member 2 10 FWD PNOTE

15 Sun Mar 20 19:13:59 2022 Member 2 -> Member 1 02 Interface is down (disconnected / link down)

16 Sun Mar 20 18:22:48 2022 Member 1 -> Member 2 03 Interface eth1-02.814 is down (disconnected / link down)

17 Tue Nov 30 18:00:45 2021 Member 2 -> Member 1 03 Interface is down (disconnected / link down)

18 Tue Nov 30 17:55:39 2021 Member 1 -> Member 2 04 Interface eth5 is down (disconnected / link down)

19 Tue Nov 30 17:11:01 2021 Member 2 -> Member 1 33 Interface is down (disconnected / link down)

20 Tue Nov 30 17:10:47 2021 Member 1 -> Member 2 02 Interface eth5 is down (disconnected / link down)

[Expert@pccfw1:0]#

Re: Reason for Firewall Failover

G_W_Albrecht — Thu, 15 Sep 2022 16:24:08 GMT

sk56202 - How to troubleshoot failovers in ClusterXL

sk62570: How to troubleshoot failovers in ClusterXL - Advanced Guide

Re: Reason for Firewall Failover

the_rock — Thu, 15 Sep 2022 16:56:32 GMT

It shows FWD pnote went down, that was the reason. So, from expert mode, run this command on both:

grep -i FWD /var/log/messages* and see what you get.

Andy

Re: Reason for Firewall Failover

Matlu — Fri, 27 Oct 2023 15:18:17 GMT

Hi,

Do you know what the description "Member state has been changed due to restart of the Cluster module" refers to?

I have a ClusterXL in Load Sharing mode, and there seems to have been a problem with the Cluster member, as apparently for a moment it stopped being part of the Cluster.

What is the MODULE of a Cluster? Does it refer to a "daemon/process"?

I share the output of the command "cphaprob show_failover".

[Expert@SG1:0]# cphaprob show_failover

Last member failover event:
Member in failure: Member 1
Reason: Member state has been changed due to restart of the Cluster module
Event time: Fri Oct 27 06:46:18 2023

Cluster failover count:
Failover counter: 1
Time of counter reset: Sun Jul 9 00:04:28 2023 (reboot)

Cluster failover history (last 20 failovers since reboot/reset on Fri Sep 24 12:49:47 2021):

No. Time: Transition: CPU: Reason:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Fri Oct 27 06:46:18 2023 Member 1 38 Member state has been changed due to restart of the Cluster module

Cheers. 🙂

Re: Reason for Firewall Failover

PhoneBoy — Fri, 27 Oct 2023 17:01:50 GMT

Policy installation per: https://support.checkpoint.com/results/sk/sk125152

Re: Reason for Firewall Failover

the_rock — Fri, 27 Oct 2023 17:55:22 GMT

https://support.checkpoint.com/results/sk/sk125152

also run -> grep -i DPWN /var/log/messages* and look for any lines for that date/time

Andy

Re: Reason for Firewall Failover

Ilya_Yusupov — Fri, 27 Oct 2023 19:55:00 GMT

Hi @ESpataro ,

FWD pnote might be triggered only in case it was terminated or crashed, in general the failover will happen only after certain time post fwd termination, so my guess you probably have a core dump, since post crash the creation of core takes a bit time which passing the time for failover trigger.

Thanks,

Ilya

Re: Reason for Firewall Failover

Matlu — Fri, 27 Oct 2023 20:08:20 GMT

Hello,

If a Failover occurs, and as a result of this a "Core DUMP" is "created", in which path is this type of files?

These files are "understandable" for common users like us, or they must be checked by Check Point TAC?

Greetings.

Re: Reason for Firewall Failover

PhoneBoy — Fri, 27 Oct 2023 23:26:22 GMT

Usermode processes should dump core in /var/log/dump/usermode
If the kernel crashes, it should be in /var/log/crash

To analyze a core dump of either kind, you will most likely need unstripped binaries and/or libraries.
These are not made available outside of R&D and TAC CFG.
Even if you have access to these things, you're going to have to involve TAC to get the underlying issue resolved with R&D.

Re: Reason for Firewall Failover

Timothy_Hall — Sat, 28 Oct 2023 14:44:16 GMT

As others have said, fwd probably crashed. Look in $FWDIR/log/fwd.elg for clues and run cpwd_admin list to confirm that the fwd process was automatically restarted and when it occurred.