Reason for Firewall Failover

[ESpataro]

Can someone please explain what the  FWD PNOTE means for failover and why it was triggered 

Details below

[Expert@pccfw1:0]# cphaprob state

Cluster Mode:   High Availability (Active Up) with IGMP Membership

ID         Unique Address  Assigned Load   State          Name                                                                                 

1 (local)  169.254.22.1    0%              STANDBY        pccfw1

2          169.254.22.2    100%            ACTIVE         pccfw2

Active PNOTEs: None

Last member state change event:

   Event Code:                 CLUS-114802

   State change:               DOWN -> STANDBY

   Reason for state change:    There is already an ACTIVE member in the cluster                                                                (member 2)

   Event time:                 Tue Sep 13 10:35:56 2022

Last cluster failover event:

   Transition to new ACTIVE:   Member 1 -> Member 2

   Reason:                     FWD PNOTE

   Event time:                 Tue Sep 13 10:35:26 2022

Cluster failover count:

   Failover counter:           45

   Time of counter reset:      Wed Aug 18 14:04:42 2021 (reboot)

[Expert@pccfw1:0]# cphaprob show_failover

Last cluster failover event:

   Transition to new ACTIVE:   Member 1 -> Member 2

   Reason:                     FWD PNOTE

   Event time:                 Tue Sep 13 10:35:26 2022

Cluster failover count:

   Failover counter:           45

   Time of counter reset:      Wed Aug 18 14:04:42 2021 (reboot)

Cluster failover history (last 20 failovers since reboot/reset on Wed Aug 18 14:04:42 2021):

 No. Time:                     Transition:           CPU:  Reason:

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

 1   Tue Sep 13 10:35:26 2022  Member 1 -> Member 2  22    FWD PNOTE

 2   Fri Sep  2 15:08:50 2022  Member 2 -> Member 1  13    ADMIN_DOWN PNOTE

 3   Tue Aug 30 19:54:20 2022  Member 1 -> Member 2  05    FWD PNOTE

 4   Sat Jun 18 11:10:51 2022  Member 2 -> Member 1  05    ADMIN_DOWN PNOTE

 5   Fri Jun 17 22:12:24 2022  Member 1 -> Member 2  04    Interface eth3.29 is down (disconnected / link down)

 6   Wed May 25 09:17:41 2022  Member 2 -> Member 1  30    ADMIN_DOWN PNOTE

 7   Wed May 25 09:05:01 2022  Member 1 -> Member 2  18    Interface eth1-03 is down (Cluster Control Protocol packets are not received)

 8   Tue May 24 23:49:22 2022  Member 2 -> Member 1  04    ADMIN_DOWN PNOTE

 9   Tue May 24 23:40:38 2022  Member 2 -> Member 1  04    Member state has been changed after returning from ACTIVE/ACTIVE scenario (remote cluster member 1 has higher priority)

 10  Tue May 24 23:30:05 2022  Member 2 -> Member 1  03    Member state has been changed after returning from ACTIVE/ACTIVE scenario (remote cluster member 1 has higher priority)

 11  Tue May 24 23:30:04 2022  Member 1 -> Member 2  26    Interface is down (disconnected / link down)

 12  Tue May 24 23:30:00 2022  Member 2 -> Member 1  24    Interface eth5 is down (disconnected / link down)

 13  Tue May 24 23:29:47 2022  Member 1 -> Member 2  04    Interface is down (disconnected / link down)

 14  Mon May  9 16:34:47 2022  Member 1 -> Member 2  10    FWD PNOTE

 15  Sun Mar 20 19:13:59 2022  Member 2 -> Member 1  02    Interface is down (disconnected / link down)

 16  Sun Mar 20 18:22:48 2022  Member 1 -> Member 2  03    Interface eth1-02.814 is down (disconnected / link down)

 17  Tue Nov 30 18:00:45 2021  Member 2 -> Member 1  03    Interface is down (disconnected / link down)

 18  Tue Nov 30 17:55:39 2021  Member 1 -> Member 2  04    Interface eth5 is down (disconnected / link down)

 19  Tue Nov 30 17:11:01 2021  Member 2 -> Member 1  33    Interface is down (disconnected / link down)

 20  Tue Nov 30 17:10:47 2021  Member 1 -> Member 2  02    Interface eth5 is down (disconnected / link down)

[Expert@pccfw1:0]#

[G_W_Albrecht]

sk56202 - How to troubleshoot failovers in ClusterXL

sk62570: How to troubleshoot failovers in ClusterXL - Advanced Guide

[the_rock]

It shows FWD pnote went down, that was the reason. So, from expert mode, run this command on both:

grep -i FWD /var/log/messages* and see what you get.

Andy

[Matlu]

Hi,

Do you know what the description "Member state has been changed due to restart of the Cluster module" refers to?

I have a ClusterXL in Load Sharing mode, and there seems to have been a problem with the Cluster member, as apparently for a moment it stopped being part of the Cluster.

What is the MODULE of a Cluster? Does it refer to a "daemon/process"?

I share the output of the command "cphaprob show_failover".

[Expert@SG1:0]# cphaprob show_failover

Last member failover event:
Member in failure: Member 1
Reason: Member state has been changed due to restart of the Cluster module
Event time: Fri Oct 27 06:46:18 2023

Cluster failover count:
Failover counter: 1
Time of counter reset: Sun Jul 9 00:04:28 2023 (reboot)

Cluster failover history (last 20 failovers since reboot/reset on Fri Sep 24 12:49:47 2021):

No. Time: Transition: CPU: Reason:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Fri Oct 27 06:46:18 2023 Member 1 38 Member state has been changed due to restart of the Cluster module

Cheers. 🙂

[PhoneBoy]

Policy installation per: https://support.checkpoint.com/results/sk/sk125152 

[the_rock]

https://support.checkpoint.com/results/sk/sk125152

also run -> grep -i DPWN /var/log/messages* and look for any lines for that date/time

Andy

[Ilya_Yusupov]

Hi @ESpataro ,

FWD pnote might be triggered only in case it was terminated or crashed, in general the failover will happen only after certain time post fwd termination, so my guess you probably have a core dump, since post crash the creation of core takes a bit time which passing the time for failover trigger.

Thanks,

Ilya

[Matlu]

Hello,

If a Failover occurs, and as a result of this a "Core DUMP" is "created", in which path is this type of files?

These files are "understandable" for common users like us, or they must be checked by Check Point TAC?

Greetings.

[PhoneBoy]

Usermode processes should dump core in /var/log/dump/usermode
If the kernel crashes, it should be in /var/log/crash

To analyze a core dump of either kind, you will most likely need unstripped binaries and/or libraries.
These are not made available outside of R&D and TAC CFG. 
Even if you have access to these things, you're going to have to involve TAC to get the underlying issue resolved with R&D.

[Timothy_Hall]

As others have said, fwd probably crashed.  Look in $FWDIR/log/fwd.elg for clues and run cpwd_admin list to confirm that the fwd process was automatically restarted and when it occurred.