FWD process does not work after reboot

[Railx]

Good afternoon.
14 days ago we upgraded to R81.10 JHF Take 87 from version R81. Today a scheduled reboot was performed, after which we are seeing issues with the FWD process.

In /var/log messages:

Jul 13 09:50:11 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE ON

Jul 13 09:50:11 2023 XXXXXX fwk: CLUS-111200-2: State change: STANDBY -> DOWN | Reason: FWD PNOTE

Jul 13 09:50:41 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE OFF

Jul 13 09:50:41 2023 XXXXXX fwk: CLUS-114802-2: State change: DOWN -> STANDBY | Reason: There is already an ACTIVE member in the cluster (member 1)

Jul 13 09:50:44 2023 XXXXXX kernel: fwd[3628]: segfault at 2d39412d ip 000000002d39412d sp 00000000fffeede0 error 14

Jul 13 09:51:16 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE ON

Jul 13 09:51:16 2023 XXXXXX fwk: CLUS-111200-2: State change: STANDBY -> DOWN | Reason: FWD PNOTE

Jul 13 09:51:46 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE OFF

Jul 13 09:51:46 2023 XXXXXX fwk: CLUS-114802-2: State change: DOWN -> STANDBY | Reason: There is already an ACTIVE member in the cluster (member 1)

Jul 13 09:51:49 2023 XXXXXX kernel: fwd[4427]: segfault at 2d39412d ip 000000002d39412d sp 00000000ff9b1fd0 error 14

Rebooting and stopping processes again did not help.

Maybe someone has some thoughts on this.

[Chris_Atkinson]

Newer JHF takes have potentially relevant FWD fixes but please consult TAC to validate accordingly. 

[the_rock]

Just to make sure I understand...are you saying upgrade was done 14 days ago, but NOT rebooted back then?

Andy

[Railx]

After the update, there was an automatic reboot and everything was fine. Nothing was manually rebooted after that.

Rail

[the_rock]

K, got it. Reason I asked was because it does give an option to uncheck an auto reboot, and I know some people do that (not recommended, but anyway). Okay...have you done any fwd debugs at all?

By the way, IF you do below and still cant figure it out,I would contact TAC right away. 

https://support.checkpoint.com/results/sk/sk86321

1. Connect to the command line on the Security Gateway / Security Management Server / StandAlone.

2. Log in to the Expert mode.

3. Stop the FWD daemon under WatchDog:

   [Expert@HostName:0]# cpwd_admin stop -name FWD
   

4. Set the debug environment variable:

   [Expert@HostName:0]# export TDERROR_ALL_ALL=5
   [Expert@HostName:0]# echo $TDERROR_ALL_ALL
   

5. Manually start the FWD daemon under debug and redirect its output to a file:

   • On a Security Gateway / StandAlone, run:

     [Expert@HostName:0]# fwd -d 1>> /var/log/fwd_debug.txt 2>> /var/log/fwd_debug.txt
     

   • On a Security Management Server, run:

     [Expert@HostName:0]# fwd -n -d 1>> /var/log/fwd_debug.txt 2>> /var/log/fwd_debug.txt
     

6. Replicate the issue.

7. Stop the FWD daemon:

   Press CTRL + C

8. Unset the debug environment variable:

   [Expert@HostName:0]# unset TDERROR_ALL_ALL
   [Expert@HostName:0]# echo $TDERROR_ALL_ALL
   

9. Start the FWD daemon under WatchDog:

   [Expert@HostName:0]# cpwd_admin start -name FWD -path "$FWDIR/bin/fwd" -command "fwd"
   [Expert@HostName:0]# cpwd_admin list | egrep "PID|FWD"
   

10. Send these files to Check Point Support for analysis:

    1. CPinfo file from the involved machine collected with the latest version of CPinfo utility from sk92739

    2. Debug output file:

       • /var/log/fwd_debug.txt

    3. Log files:

       • $FWDIR/log/fwd.elg*
       • $CPDIR/log/cpwd.elg*
       • /var/log/message*

    Note: Compress (TAR/GZIP) all the requested files BEFORE uploading them from the machine - if uploading over FTP, then it is mandatory to use BIN mode