Probably best to involve Check Point TAC to help diagnose this, but I did cover how to dig into the behavior of runaway or misbehaving processes in my book, here is a bit of it:
For certain processes there is a log file available which may prove helpful if the
process is logging any errors:
- fwd: $FWDIR/log/fwd.elg
- fwm: $FWDIR/log/fwm.elg
- cpm: $FWDIR/log/cpm.elg
- vpnd: $FWDIR/log/vpnd.elg
- cpd: $CPDIR/log/cpd.elg
For example to look at the latest entries in the fwd process log, type tail
$FWDIR/log/fwd.elg. Further debugging of runaway or misbehaving processes can
be accomplished with these commands:
- fw debug fwd on|off (Debugs go to $FWDIR/log/fwd.elg)
- fw debug fwm on|off (Debugs go to $FWDIR/log/fwm.elg)
- $FWDIR/scripts/cpm_debug.sh -t <TOPIC> -s <SEVERITY>
- vpn debug on|off (Debugs go to $FWDIR/log/vpnd.elg)
- cpd_admin debug on|off (Debugs go to $CPDIR/log/cpd.elg)
If you enable debug on a process, make sure to turn it off when done! To identify the
function of these or any other daemon process names that may be encountered, consult
sk97638: Check Point Processes and Daemons.
But suppose you see a strange-looking process hogging CPU in the top output:
Figure 4-7: A Process Consuming Excessive CPU
Huh? Why is the cat command eating so much CPU and what process launched this
beast? The built-in Gaia tool pstree can help give us the answer:
Figure 4-8: Using pstree to Help Identify a CPU-Hogging Process
As it turns out this process is related to the scrubd process which sk97638 lists as
part of the Threat Extraction blade feature. Using pstree can help us determine the
origin of a runaway process when the name of the process itself (or its arguments) is not
enough to determine its purpose and whether we should do something about it.
Another command that can provide some insight for a mystery process consuming
excessive CPU is lsof (LiSt Open Files). Once the process ID has been determined
(15215 in our example) run lsof -p 15215. The output will show all files currently
being accessed by the process, along with all open network connections and TCP/UDP
ports the process is currently listening on (if any).
A final option that is guaranteed to satisfy even the most voracious seeker of
information about a pesky process, is to install the strace binary (similar to the truss
command on Solaris) and dynamically attach it to the problematic process with the –p
option. This allows real-time monitoring of all system calls made by the process in an
attempt to determine what it is trying to accomplish. strace is not included by default
with Gaia (and its use is not officially supported) but you can install the precompiled
CentOS version of strace available here:
http://vault.centos.org/3.8/os/i386/RedHat/RPMS/strace-4.5.14-0.EL3.1.i386.rpm
