topic Re: Is there any one of this obstacle in the Checkup? in Firewall and Security Management

Is there any one of this obstacle in the Checkup?

TAEKBOM_Kim — Wed, 27 Jun 2018 02:06:35 GMT

* Customer environment

- Traffic : 1Gbps

- User : 5000

* Checkup Platform/Version

- SG15600 / Product version Check Point Gaia R80.10 - OS build 462

* The point at issue

- FWD or FWM daemon is stopped within 2 to 3 days of checkup installation

[Expert@Checkup-Demo:0]# cpwd_admin list
APP        PID    STAT #START START_TIME             MON COMMAND
CPVIEWD    22325 E     1       [15:17:58] 14/6/2018   N    cpviewd
HISTORYD   22328 E     1       [15:17:58] 14/6/2018   N    cpview_historyd
CPD        22340 E     1       [15:17:58] 14/6/2018   Y    cpd
MPDAEMON   22352 E     1       [15:17:59] 14/6/2018   N    mpdaemon /opt/CPshrd-R80/log/mpdaemon.elg /opt/CPshrd-R80/conf/mpdaemon.conf
CI_CLEANUP 22703 E     1       [15:18:06] 14/6/2018   N    avi_del_tmp_files
CIHS       22705 E     1       [15:18:06] 14/6/2018   N    ci_http_server -j -f /opt/CPsuite-R80/fw1/conf/cihs.conf
FWD        0      T     0       [09:23:45] 18/6/2018   N    fwd
FWM        22750 E     1       [15:18:07] 14/6/2018   N    fwm
CPM        22971 E     1       [15:18:09] 14/6/2018   N    /opt/CPsuite-R80/fw1/scripts/cpm.sh -s
....
...
..

[Expert@checkup-demo:0]# cpview
CPView: Failed parsing the conf file
[Expert@checkup-demo:

- FWD, FWM daemon do not run when device is restarted after first failure

[Expert@checkup-demo:0]# ps -aux | grep fw
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
admin     7387 0.0 0.0   1736   512 pts/2    S+   17:20   0:00 grep fw
admin    20857 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_0]
admin    20858 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_1]
admin    20859 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_2]
admin    20860 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_3]
admin    20861 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_4]
admin    20862 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_5]
admin    20863 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_6]
admin    20864 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_7]
admin    20865 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_8]
admin    20866 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_9]
admin    20867 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_10]
admin    20868 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_11]
admin    20869 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_12]
admin    20870 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_13]
admin    20871 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_14]
admin    20872 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_15]
admin    20873 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_16]
admin    20874 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_17]
admin    20875 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_18]
admin    20876 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_19]
admin    20877 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_20]
admin    20878 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_21]
admin    20879 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_22]
admin    20880 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_23]
admin    20881 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_24]
admin    20882 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_25]
admin    20883 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_26]
admin    20884 0.0 0.0      0     0 ?        S    11:40   0:00 [fw_worker_27]
[Expert@checkup-demo:0]#

- The same symptom occurred again after replacing the equipment.

* Questions

1. Is there any one of this obstacle in the Checkup?

2. How to Enable the Failed Equipment to run FWD and FWM Daemons? (It is useless to reboot / cpstart)

3. How to solve this problem?

Re: Is there any one of this obstacle in the Checkup?

Kaspars_Zibarts — Wed, 27 Jun 2018 07:26:50 GMT

Generally this SK is a very good source of information where to look for logs for different processes:

Check Point Processes and Daemons

From there you would find that fwm log is here $FWDIR/log/fwm.elg and fwd here $FWDIR/log/fwd.elg. Check those logs.

This SK is very helpful to explain how these processes relate to each other and how to debug them

R80.x Security Management server main processes debugging

Is this a standalone deployment? Management and gateway running on the same box?

Re: Is there any one of this obstacle in the Checkup?

Marco_Valenti — Wed, 27 Jun 2018 07:32:00 GMT

You could you cpwd_admin start command , but the issue seems to be deeper here

> cpwd_admin start -name <process name> -path "<full path>" -command
"<executable name>"
Parameter Description
-name <process name> A name for the process to be watched by WatchDog.
-path "<full path>" The full path to the executable including the executable name
-command "<executable
name>"
The name of the executable file.
Example To start and monitor the fwm process.
> cpwd_admin start -name FWM -path "$FWDIR/bin/fwm" -command "fwm"

Re: Is there any one of this obstacle in the Checkup?

PhoneBoy — Wed, 27 Jun 2018 15:12:00 GMT

How much memory is in the 15600?

R80.10 Management requires a lot more memory and it will certainly improve performance to have as much as possible.

Especially in a standalone configuration used in Security Checkups.

Also, if you haven't opened a TAC case, I recommend doing so.

Re: Is there any one of this obstacle in the Checkup?

TAEKBOM_Kim — Wed, 27 Jun 2018 23:54:56 GMT

I will check "Check Point Processes and Daemons " and "R80.x Security Management server main processes debugging"

Yes, It is management and gateway running on the same box.

Re: Is there any one of this obstacle in the Checkup?

TAEKBOM_Kim — Thu, 28 Jun 2018 01:23:37 GMT

It doesn't work.

> cpwd_admin start -name FWM -path "$FWDIR/bin/fwm" -command "fwm"

[Expert@checkup-demo:0]# cpwd_admin start -name FWM -path "$FWDIR/bin/fwm" -command "fwm"
cpwd_admin: Failed to submit request to cpWatchDog
[Expert@checkup-demo:0]# cpview
CPView: Failed parsing the conf file
[Expert@checkup-demo:0]#

Re: Is there any one of this obstacle in the Checkup?

TAEKBOM_Kim — Thu, 28 Jun 2018 01:55:26 GMT

The 15600 is installed 32GB Memory.

I will open the TAC case.

Thank you for your advice 🙂

Re: Is there any one of this obstacle in the Checkup?

Kaspars_Zibarts — Thu, 28 Jun 2018 06:01:19 GMT

Check if the watchdog is actually running

[Expert@fwf1:0]# ps aux | grep cpwd
admin 5241 0.0 0.0 1736 508 pts/2 S+ 07:57 0:00 grep cpwd
admin 20182 0.0 0.0 14544 3416 ? Ss Apr14 2:22 /opt/CPshrd-R80/bin/cpwd

If you don't see it running, try starting it manually simply by running

/opt/CPshrd-R80/bin/cpwd

I'm sure it's not related but we had one really weird case where management server failed to start after reboot as watchdog failed to start. Didn't matter how many times we rebooted it. But then after running watchdog manually, it all fixed "itself".

Re: Is there any one of this obstacle in the Checkup?

TAEKBOM_Kim — Thu, 28 Jun 2018 06:30:45 GMT

[Expert@checkup-demo:0]# ps -aux | grep cpwd
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
admin 29789 0.0 0.0 1736 524 pts/2 S+ 15:20 0:00 grep cpwd
[Expert@checkup-demo:0]# /opt/CPshrd-R80/bin/cpwd
[Expert@checkup-demo:0]# /opt/CPshrd-R80/bin/cpwd
[Expert@checkup-demo:0]# /opt/CPshrd-R80/bin/cpwd
[Expert@checkup-demo:0]# ps -aux | grep cpwd
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
admin 29801 0.0 0.0 1732 520 pts/2 S+ 15:20 0:00 grep cpwd
[Expert@checkup-demo:0]# cpwd_admin start -name FWM -path "$FWDIR/bin/fwm" -command "fwm"
cpwd_admin: Failed to submit request to cpWatchDog
[Expert@checkup-demo:0]# cpview
CPView: Failed parsing the conf file
[Expert@checkup-demo:0]#

After FWD or FWM daemon is stopped within 2 to 3 days of checkup installation.

I tried to reboot the 15600 but both FWD and FWM don't run.

it is weird.

Re: Is there any one of this obstacle in the Checkup?

Marco_Valenti — Thu, 28 Jun 2018 07:10:13 GMT

This definitely look something for the Tac

Re: Is there any one of this obstacle in the Checkup?

Bruno_Duarte — Thu, 28 Jun 2018 10:10:52 GMT

Isn´t that related with the sk 105510?

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk105510

I faced a similar issue in 2 security Checkups in the same week, I saw that there were more people complainning in a mailing list

Re: Is there any one of this obstacle in the Checkup?

TAEKBOM_Kim — Fri, 29 Jun 2018 01:01:01 GMT

Thank you so much 🙂
It helped me.

You faced a similar issue in 2 security Checkups.
Did you configure a standalone deployment(security management and security gateway running on the same box)?

Re: Is there any one of this obstacle in the Checkup?

Bruno_Duarte — Fri, 29 Jun 2018 08:58:01 GMT

Yes, it appears it is happenning in standalone deployments (to me, always with take 103), this week it happenned again to a team mate, and the stange it is that the issue re-occurres..