CPD-process going high on active member after up-gradation of a normal cluster

[Sahil_Qos]

Hi all, We have recently upgraded a cluster to GAIA R80.10, but after updgradation CPD process is going high on the active member causing disconnection with the manager and we are not able to push policy.

[Marco_Valenti]

A bit more of info on your environment would be nice , do you have already installed the jhf take 70 for r80.10 ?

[Sahil_Qos]

Hi Marco,

JHF take_70 already installed on the Gateways.

[Timothy_Hall]

Is the cpd process just consuming large amounts of CPU or is it constantly crashing and restarting?  Use cpwd_admin list to determine the latter, and take a look in $CPDIR/log/cpd.elg to figure out the former.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Sahil_Qos]

Hi Tim,

the cpd process is consuming more than 95% of CPU in production hours and fluctuating in lean-hours, no core-dumps generated. Even raised a SR with TAC for this issue. Followed some sk's as well as per the error-message in cpd.elg but still no luck.

[Timothy_Hall]

Probably best to involve Check Point TAC to help diagnose this, but I did cover how to dig into the behavior of runaway or misbehaving processes in my book, here is a bit of it:

For certain processes there is a log file available which may prove helpful if the
process is logging any errors:

• fwd: $FWDIR/log/fwd.elg
• fwm: $FWDIR/log/fwm.elg
• cpm: $FWDIR/log/cpm.elg
• vpnd: $FWDIR/log/vpnd.elg
• cpd: $CPDIR/log/cpd.elg

For example to look at the latest entries in the fwd process log, type tail
$FWDIR/log/fwd.elg. Further debugging of runaway or misbehaving processes can
be accomplished with these commands:

• fw debug fwd on|off (Debugs go to $FWDIR/log/fwd.elg)
• fw debug fwm on|off (Debugs go to $FWDIR/log/fwm.elg)
• $FWDIR/scripts/cpm_debug.sh -t <TOPIC> -s <SEVERITY>
• vpn debug on|off (Debugs go to $FWDIR/log/vpnd.elg)
• cpd_admin debug on|off (Debugs go to $CPDIR/log/cpd.elg)

If you enable debug on a process, make sure to turn it off when done! To identify the
function of these or any other daemon process names that may be encountered, consult
sk97638: Check Point Processes and Daemons.

But suppose you see a strange-looking process hogging CPU in the top output:
Figure 4-7: A Process Consuming Excessive CPU

Huh? Why is the cat command eating so much CPU and what process launched this
beast? The built-in Gaia tool pstree can help give us the answer:
Figure 4-8: Using pstree to Help Identify a CPU-Hogging Process

As it turns out this process is related to the scrubd process which sk97638 lists as
part of the Threat Extraction blade feature. Using pstree can help us determine the
origin of a runaway process when the name of the process itself (or its arguments) is not
enough to determine its purpose and whether we should do something about it.

Another command that can provide some insight for a mystery process consuming
excessive CPU is lsof (LiSt Open Files). Once the process ID has been determined
(15215 in our example) run lsof -p 15215. The output will show all files currently
being accessed by the process, along with all open network connections and TCP/UDP
ports the process is currently listening on (if any).

A final option that is guaranteed to satisfy even the most voracious seeker of
information about a pesky process, is to install the strace binary (similar to the truss
command on Solaris) and dynamically attach it to the problematic process with the –p
option. This allows real-time monitoring of all system calls made by the process in an
attempt to determine what it is trying to accomplish. strace is not included by default
with Gaia (and its use is not officially supported) but you can install the precompiled
CentOS version of strace available here:

http://vault.centos.org/3.8/os/i386/RedHat/RPMS/strace-4.5.14-0.EL3.1.i386.rpm

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com