topic FWD process does not work after reboot in General Topics

FWD process does not work after reboot

Railx — Thu, 13 Jul 2023 08:22:27 GMT

Good afternoon.
14 days ago we upgraded to R81.10 JHF Take 87 from version R81. Today a scheduled reboot was performed, after which we are seeing issues with the FWD process.

In /var/log messages:

Jul 13 09:50:11 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE ON

Jul 13 09:50:11 2023 XXXXXX fwk: CLUS-111200-2: State change: STANDBY -> DOWN | Reason: FWD PNOTE

Jul 13 09:50:41 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE OFF

Jul 13 09:50:41 2023 XXXXXX fwk: CLUS-114802-2: State change: DOWN -> STANDBY | Reason: There is already an ACTIVE member in the cluster (member 1)

Jul 13 09:50:44 2023 XXXXXX kernel: fwd[3628]: segfault at 2d39412d ip 000000002d39412d sp 00000000fffeede0 error 14

Jul 13 09:51:16 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE ON

Jul 13 09:51:16 2023 XXXXXX fwk: CLUS-111200-2: State change: STANDBY -> DOWN | Reason: FWD PNOTE

Jul 13 09:51:46 2023 XXXXXX fwk: CLUS-120100-2: fwd PNOTE OFF

Jul 13 09:51:46 2023 XXXXXX fwk: CLUS-114802-2: State change: DOWN -> STANDBY | Reason: There is already an ACTIVE member in the cluster (member 1)

Jul 13 09:51:49 2023 XXXXXX kernel: fwd[4427]: segfault at 2d39412d ip 000000002d39412d sp 00000000ff9b1fd0 error 14

Rebooting and stopping processes again did not help.

Maybe someone has some thoughts on this.

Re: FWD process does not work after reboot

Chris_Atkinson — Thu, 13 Jul 2023 12:31:48 GMT

Newer JHF takes have potentially relevant FWD fixes but please consult TAC to validate accordingly.

Re: FWD process does not work after reboot

the_rock — Thu, 13 Jul 2023 13:41:44 GMT

Just to make sure I understand...are you saying upgrade was done 14 days ago, but NOT rebooted back then?

Andy

Re: FWD process does not work after reboot

Railx — Fri, 14 Jul 2023 10:06:25 GMT

After the update, there was an automatic reboot and everything was fine. Nothing was manually rebooted after that.

Rail

Re: FWD process does not work after reboot

the_rock — Fri, 14 Jul 2023 10:32:38 GMT

K, got it. Reason I asked was because it does give an option to uncheck an auto reboot, and I know some people do that (not recommended, but anyway). Okay...have you done any fwd debugs at all?

By the way, IF you do below and still cant figure it out,I would contact TAC right away.

https://support.checkpoint.com/results/sk/sk86321

Connect to the command line on the Security Gateway / Security Management Server / StandAlone.
Log in to the Expert mode.

Stop the FWD daemon under WatchDog:

[Expert@HostName:0]# cpwd_admin stop -name FWD

Set the debug environment variable:

[Expert@HostName:0]# export TDERROR_ALL_ALL=5
[Expert@HostName:0]# echo $TDERROR_ALL_ALL

Manually start the FWD daemon under debug and redirect its output to a file:

On a Security Gateway / StandAlone, run:

[Expert@HostName:0]# fwd -d 1>> /var/log/fwd_debug.txt 2>> /var/log/fwd_debug.txt

On a Security Management Server, run:

[Expert@HostName:0]# fwd -n -d 1>> /var/log/fwd_debug.txt 2>> /var/log/fwd_debug.txt

Replicate the issue.
Stop the FWD daemon:
Press CTRL + C

Unset the debug environment variable:

[Expert@HostName:0]# unset TDERROR_ALL_ALL
[Expert@HostName:0]# echo $TDERROR_ALL_ALL

Start the FWD daemon under WatchDog:

[Expert@HostName:0]# cpwd_admin start -name FWD -path "$FWDIR/bin/fwd" -command "fwd"
[Expert@HostName:0]# cpwd_admin list | egrep "PID|FWD"

Send these files to Check Point Support for analysis:
1. CPinfo file from the involved machine collected with the latest version of CPinfo utility from sk92739
2. Debug output file:
  - /var/log/fwd_debug.txt
3. Log files:
  - $FWDIR/log/fwd.elg*
  - $CPDIR/log/cpwd.elg*
  - /var/log/message*
Note: Compress (TAR/GZIP) all the requested files BEFORE uploading them from the machine - if uploading over FTP, then it is mandatory to use BIN mode