cancel
Showing results for 
Search instead for 
Did you mean: 
Post a Question
ED
Silver

cpwd_admin list overview (SMS)

cpwd_admin list command is mentioned in the thread top 3 CLI commands and is an essential command to know to quickly check that key processes are up and running. I think it's also nice to know what each process are responsible for. RFL, room buddies for life? If you have a standalone installation you can prevent downtime by knowing what to restart and avoid cpstop/cpstart/reboot. 

This shows an example from a security managment server. On a security gateway some of these will also be there but others in addition. If you take a closer look you will see a process called LPD which have another start date/time and nowhere to find what this process do. Can someone explain me what is LPD? I can not find documentation for this process. 

Important to understand each column and its value. 

Column numberExplanation
1APP. Application. Name of process. 
2PID (Process identification number). 
3STAT (status). E-established. T-terminated.
4#START. How many times the process has started since cpwd took control of the process.
5START_TIME. The last time the process started.
6MON. Monitored actively. YES/NO. 
7Command. Used by cpwd to start the process. 

STAT column should have every row with the value E-established, meaning that it's running. If the value is T-terminated you should start the process and find out why it crashed/won't start. #START shows how many times the process has started. The values should be 1 and if the value is higher than 1 then something has happened with that process, causing restart and the value to increase. Also the start time should be very close to the other processes and not so far away from the time server booted up. We must mention cpwd (Check Point Watchdog daemon) which is a process that launches and monitors critical processes such as Check Point daemons on the local machine, and attempts to restart them if they fail.

Do you know what each process does? What happens if it's terminated? How to start/stop? How to debug?

Following is an explanation for each process from this example above (except lpd). From Check Point:

cpviewd:

DescriptionOn Security Gateway and Management Server.
CPView Utility daemon (sk101878).
Path
  • In R77.30 and above:
    $CPDIR/bin/cpviewd
  • In R77-R77.20:
    $FWDIR/bin/cpviewd
Configuration file$CPDIR/conf/cpview_conf.xml
Notes"cpwd_admin list" command shows the process as "CPVIEWD".
To stop[Expert@HostName]# cpwd_admin stop -name CPVIEWD
To start
  • In R77.30 and above:
    [Expert@HostName]# cpwd_admin start -name CPVIEWD -path "$CPDIR/bin/cpviewd" -command "cpviewd" 

  • In R77-R77.20:
    [Expert@HostName]# cpwd_admin start -name CPVIEWD -path "$FWDIR/bin/cpviewd" -command "cpviewd"
DebugRefer to sk101878

cpd:

Description
  • Port 18191 - Generic process (add-ons container) for many Check Point services, such as installing and fetching policy, and online updates
  • Port 18211 - SIC push certificate (from Internal CA)
Path$CPDIR/bin/cpd
%CPDIR%\bin\cpd
Logfile$CPDIR/log/cpd.elg
%CPDIR%\log\cpd.elg
Notes"cpwd_admin list" command shows the process as "CPD".
To stop
  • MGMT / Gateway mode:

    [Expert@HostName]# cpwd_admin stop -name CPD -path "$CPDIR/bin/cpd_admin" -command "cpd_admin stop"
    or
    [Expert@HostName]# cpstop 

  • VSX mode:

    [Expert@HostName:0]# vsenv <VSID>
    [Expert@HostName:<VSID>]# cpwd_admin stop -name CPD -ctx <VSID> -path "$CPDIR/bin/cpd_admin" -command "cpd_admin stop" -env inherit 
    or
    [Expert@HostName:0]# cpstop
To start
  • MGMT / Gateway mode:

    [Expert@HostName]# cpwd_admin start -name CPD -path "$CPDIR/bin/cpd" -command "cpd"
    or
    [Expert@HostName]# cpstart 

  • VSX mode:

    [Expert@HostName:0]# vsenv <VSID>
    [Expert@HostName:<VSID>]# cpwd_admin start -name CPD -ctx <VSID> -path "$CPDIR/bin/cpd" -command "cpd" -env inherit 
    or
    [Expert@HostName:0]# cpstart
Debug"cpd_admin debug" - refer to sk86320

fwd:

Description
  • Logging
  • Spawning child processes (e.g., vpnd)
Path$FWDIR/bin/fwd
%FWDIR%\bin\fwd
Logfile$FWDIR/log/fwd.elg
%FWDIR%\log\fwd.elg
Notes
  • "cpwd_admin list" command shows the process as "FWD".
  • "top" / "ps" commands might also show "fw" process and/or "fw_full" process, which are just wrappers for the "fwd" process.
To stop
  • MGMT / Gateway mode:

    [Expert@HostName]# cpwd_admin stop -name FWD -path "$FWDIR/bin/fw" -command "fw kill fwd"
    or
    [Expert@HostName]# cpstop 

  • VSX mode:

    [Expert@HostName:0]# vsenv <VSID>
    [Expert@HostName:<VSID>]# cpwd_admin stop -name FWD -ctx <VSID> -path "$FWDIR/bin/fw" -command "fw kill fwd" -env inherit
    or
    [Expert@HostName:0]# cpstop
To start
  • MGMT / Gateway mode:

    [Expert@HostName]# cpwd_admin start -name FWD -path "$FWDIR/bin/fw" -command "fwd"
    or
    [Expert@HostName]# cpstart 

  • VSX mode:

    [Expert@HostName:0]# vsenv <VSID>
    [Expert@HostName:<VSID>]# cpwd_admin start -name FWD -ctx <VSID> -path "$FWDIR/bin/fwd" -command "fwd" -env inherit
    or
    [Expert@HostName:0]# cpstart
DebugRefer to sk86321
  1. Start debug:
    fw debug fwd on TDERROR_ALL_ALL=5
    fw debug fwd on OPSEC_DEBUG_LEVEL=3
  2. Replicate the issue
  3. Stop debug:
    fw debug fwd off TDERROR_ALL_ALL=0
    fw debug fwd off OPSEC_DEBUG_LEVEL=0
  4. Analyze:
    $FWDIR/log/fwd.elg*

fwm:

Description

Communication between SmartConsole applications and Security Management Server.

Path$FWDIR/bin/fwm
%FWDIR%\bin\fwm
Logfile$FWDIR/log/fwm.elg
%FWDIR%\log\fwm.elg
Notes"cpwd_admin list" command shows the process as "FWM".
To stop

[Expert@HostName]# cpwd_admin stop -name FWM -path "$FWDIR/bin/fwm" -command "fw kill fwm"

In addition, on R8x, you can use the ngm_stop.sh script (refer to sk111772😞
  • $FWDIR/scripts/ngm_stop.sh
    (refer to $FWDIR/log/ngm_stop.elg)
  • $MDS_TEMPLATE/scripts/ngm_stop.sh
    (refer to $MDS_TEMPLATE/log/ngm_stop.elg)
To start

[Expert@HostName]# cpwd_admin start -name FWM -path "$FWDIR/bin/fwm" -command "fwm"

In addition, on R8x, you can use the ngm_start.sh script (refer to sk111772😞
  • $FWDIR/scripts/ngm_start.sh
    (refer to $FWDIR/log/ngm_start.elg)
  • $MDS_TEMPLATE/scripts/ngm_start.sh
    (refer to $MDS_TEMPLATE/log/ngm_start.elg)
Debug
  • Security Management Server - refer to sk86186:

    1. Start debug:
      fw debug fwm on TDERROR_ALL_ALL=5
      fw debug fwm on OPSEC_DEBUG_LEVEL=3
    2. Replicate the issue
    3. Stop debug:
      fw debug fwm off TDERROR_ALL_ALL=0
      fw debug fwm off OPSEC_DEBUG_LEVEL=0
    4. Analyze:
      $FWDIR/log/fwm.elg*
  • Domain Management Server - refer to sk33207:

    1. Switch to the context of the relevant Domain Management Server:
      mdsenv <Domain_Name>
    2. Start debug:
      fw debug fwm on TDERROR_ALL_ALL=5
      fw debug fwm on OPSEC_DEBUG_LEVEL=3
    3. Replicate the issue
    4. Stop debug:
      fw debug fwm off TDERROR_ALL_ALL=0
      fw debug fwm off OPSEC_DEBUG_LEVEL=0
    5. Analyze:
      $FWDIR/log/fwm.elg*
  • Multi-Domain Security Management Server - refer to sk33208:

    1. Start debug:
      fw debug mds on TDERROR_ALL_ALL=5
      fw debug mds on OPSEC_DEBUG_LEVEL=3
    2. Replicate the issue
    3. Stop debug:
      fw debug mds off TDERROR_ALL_ALL=0
      fw debug mds off OPSEC_DEBUG_LEVEL=0
    4. Analyze:
      $MDS_TEMPLATE/log/mds.elg*

SOLR (java_solr):

Description

Starting in R80 (SmartEvent NGSE was integrated).

Jetty Server.

Events are stored in the SOLR database.

Path$RTDIR/bin/java_solr
Logfile$RTDIR/log/solr.log
$RTDIR/log/solrRun.log
Notes""cpwd_admin list" command shows the process as "SOLR".
Configuration$RTDIR/conf/jetty.xml
$RTDIR/conf/solr.log4j.properties
$RTDIR/conf/solrConnectionConfig.xml
$RTDIR/log_indexes/solr.xml
To stop

[Expert@HostName]# evstop

To start

[Expert@HostName]# evstart

Debug

Refer to sk105806.

SmartEventSetDebugLevel solr <debug_level>

$FWDIR/scripts/solr_debug.py {on | off}

RFL (LogCore):

Description

Starting in R80 (SmartEvent NGSE was integrated).

Manages the queries it gets from the consumer processes, forwards them to SOLR database and returns the results. Also in charge of resolving and database maintenance (clean up old indexes to have space for the new ones).

Path$RTDIR/bin/LogCore
Logfile$RTDIR/log/RFL.log
$RTDIR/log/rflRun.log
Notes"cpwd_admin list" command shows the process as "RFL".
Configuration$RTDIR/conf/rfl.log4j.properties
$RTDIR/conf/rfl.log4j.properties.forUpgrade
$RTDIR/conf/rflConfig.xml
To stop

[Expert@HostName]# evstop

To start

[Expert@HostName]# evstart

Debug

Refer to sk105806.

SmartEventSetDebugLevel rfl <debug_level>

SmartView:

Description

SmartEvent Web Application that allows you to connect to SmartEvent NGSE server (at https://<IP_Address_of_SmartEvent_Server>/smartview/) and see the event views and analysis directly from a Web Browser, without installing SmartConsole. The Web page comes with predefined views that you can customize.

Refer to sk105684.

Path$RTDIR/bin/SmartView
Logfile$RTDIR/log/smartview.log
$RTDIR/log/SmartViewRun.log
$RTDIR/log/smartview-service.log
Notes"cpwd_admin list" command shows the process as "SMARTVIEW"
Configuration$RTDIR/conf/smartview.log4j.properties
To stop

[Expert@HostName]# evstop

To start

[Expert@HostName]# evstart

Debug

Refer to sk105806.

SmartEventSetDebugLevel smartview <debug_level>

Indexer (log_indexer):

Description

Starting in R80 (SmartEvent NGSE was integrated).

Log indexer.

Path$RTDIR/log_indexer/log_indexer
Logfile$RTDIR/log_indexer/log/log_indexer.elg
$RTDIR/log_indexer/log/log_indexerRun.log
Notes"cpwd_admin list" command shows the process as "INDEXER".
Configuration$RTDIR/log_indexer/conf/log_indexer_settings.conf
$RTDIR/log_indexer/log_indexer_custom_settings.conf
To stop

[Expert@HostName]# evstop

To start

[Expert@HostName]# evstart

CPM:

Description

On Security Management Server R80 and above:

  • Serves requests from SmartConsole
  • Responsible for writing all information to the PostgreSQL and SOLR databases
Path$FWDIR/scripts/cpm.sh
Logfile$FWDIR/log/cpm.elg
Notes"cpwd_admin list" command shows the process as "CPM".
To stop

[Expert@HostName]# cpstop

In addition, on R8x, you can use the ngm_stop.sh script (refer to sk111772😞

  • $FWDIR/scripts/ngm_stop.sh
    (refer to $FWDIR/log/ngm_stop.elg)
  • $MDS_TEMPLATE/scripts/ngm_stop.sh
    (refer to $MDS_TEMPLATE/log/ngm_stop.elg)
To start

[Expert@HostName]# cpstart

In addition, on R8x, you can use the ngm_start.sh script (refer to sk111772😞

  • $FWDIR/scripts/ngm_start.sh
    (refer to $FWDIR/log/ngm_start.elg)
  • $MDS_TEMPLATE/scripts/ngm_start.sh
    (refer to $MDS_TEMPLATE/log/ngm_start.elg)
Debug

Refer to sk115557

SMARTLOG_SERVER:

Description

SmartLog product.
Path$SMARTLOGDIR/smartlog_server
Logfile$SMARTLOGDIR/log/smartlog_server.elg
Notes"cpwd_admin list" command shows the process as "SMARTLOG_SERVER"
To stop

[Expert@HostName]# smartlogstop

To start

[Expert@HostName]# smartlogstart

Debug
  1. Stop SmartLog:
    smartlogstop
  2. Start SmartLog under debug:
    env TDERROR_ALL_ALL=5 $SMARTLOGDIR/smartlog_server 1>> /var/log/smartlog.debug 2>> /var/log/smartlog.debug
  3. Replicate the issue
  4. Stop debug - press CTRL+C.
  5. Start SmartLog normally:
    smartlogstart

DAService:

Description

Check Point Upgrade Service Engine (CPUSE) - former 'Gaia Software Updates' service (refer to sk92449).

Path$DADIR/bin/DAService
Logfile/opt/CPInstLog/DeploymentAgent.log
/opt/CPInstLog/DA_UI.log
Notes"cpwd_admin list" command shows the process as "DASERVICE"
(command is "$DADIR/bin/DAService_script" - this is a watchdog script that starts the $DADIR/bin/DAService if it is not running).
To stop
  1. [Expert@HostName]# $DADIR/bin/dastop
  2. [Expert@HostName]# dbget installer:stop
To start
  1. [Expert@HostName]# $DADIR/bin/dastart
  2. [Expert@HostName]# dbget installer:start
Debug

Refer to sk92449:

  1. Create the configuration file:
    touch $DADIR/bin/DAconf
  2. Add the following line (case-sensitive; spaces are not allowed):
    PING_TRACE=1
  3. Save the changes
  4. Re-load the new configuration:
    DAClient conf
  5. As soon as possible:
    1. Replicate the issue
    2. Delete the $DADIR/bin/DAconf file
    3. Re-load the configuration with DAClient conf command
  6. Analyze:
    /opt/CPInstLog/DeploymentAgent.log

CPSM (cpstat_monitor):

Description

Process is responsible for collecting and sending information to SmartView Monitor.

Path$FWDIR/bin/cpstat_monitor
%FWDIR%\bin\cpstat_monitor
Logfile$FWDIR/log/cpstat_monitor.elg
%FWDIR%\log\cpstat_monitor.elg
Notes
  • "cpwd_admin list" command shows the process as "CPSM".
  • By default, does not run in the context of Domain Management Servers.
  • By default, in MGMT HA runs only on "Active" Security Management Server.
Configuration$RTDIR/conf/jetty.xml
$RTDIR/conf/solr.log4j.properties
$RTDIR/conf/solrConnectionConfig.xml
$RTDIR/log_indexes/solr.xml
To stop

[Expert@HostName]# cpwd_admin stop -name CPSM

To start

[Expert@HostName]# cpwd_admin start -name CPSM -path "$FWDIR/bin/cpstat_monitor" -command "cpstat_monitor"

Debug

Refer to sk108177

11 Replies

Re: cpwd_admin list overview (SMS)

I can see a hint:

/opt/CPdiag-R80.20/bin/lpd

/opt/CPdiag-R80.20/bin/cpdiag

 # cpdiag --h
Log path: /opt/CPsuite-R80.20/fw1/log/cpdiag.elg
CPDiag options:
  -f [ --file ] arg     XML files to upload
  -v [ --version ]      Display the CPDiag version number
  -h [ --help ]         Display this help and exit
  -A [ --Add ]          Add user defined data. Must be paired with --Key and --Val
  -R [ --Remove ]       Removes a user defined data key. Must be paired with --Key
  -P [ --Print ]        Displays the user defined data
  -K [ --Key ] arg      Identifies a user created data entry
  -V [ --Val ] arg      The content of a user created data entry

#  cpdiag -P
Log path: /opt/CPsuite-R80.20/fw1/log/cpdiag.elg
The entered custom attributes are:

Context: single_context
        lpd_signatures_hash=c545fc641f5f1d647b0d66c0501f9728
        signatures_hash=e75d4aa314a8e972c2d233689e6c3294

ED
Silver

Re: cpwd_admin list overview (SMS)

Thank you for leading me to CPdiag but I still don't understand what lpd does. 

ED
Silver

Re: cpwd_admin list overview (SMS)

pstree shows this:

And also:

0 Kudos

Re: cpwd_admin list overview (SMS)

Thanks for sharing, nice post. When I check form the management servers (R80.10), I realised that one of them has an LPD but the other one does not ? I could not find any information about this daemon.

ED
Silver

Re: cpwd_admin list overview (SMS)

It's weird that there is no documentation about lpd. 

0 Kudos
Employee+
Employee+

Re: cpwd_admin list overview (SMS)

Hi,

The LPD (or Log Parser Daemon) will scan preconfigured files and search for predefined signatures.
CPDiag will use those results later on.

TBH I don't know too much about this daemon and how it works, but Günther had the right idea when he pointed you to CPDiag.

HTH

 Yonatan 

Re: cpwd_admin list overview (SMS)

Hi Enis,

Thanks for a great article.

I have a question, for you or anyone out there who might know.

It is about the MON column in the cpwd_admin list command.

If I put it to the test by killing the fwd on the management server (kill <pid of fwd>) and waiting and watching then I see that it automatically gets started up again and the new PID is shown and the start count goes up by 1 (within the allowed 60 seconds).

The strange part is that fwd is marked as N for MON.

This goes against my understanding of the monitoring and that N meant that something that was Not monitored would not be restarted.

Is there an explanation for this or something that I am missing?

Don

0 Kudos
ED
Silver

Re: cpwd_admin list overview (SMS)

Hi Donald, 

Take a look at this thread at the end and you will get an answer to your question. 

https://community.checkpoint.com/thread/5201-add-a-process-to-be-monitored-by-cpwatchdog

0 Kudos

Re: cpwd_admin list overview (SMS)

Hi Enis,

I read the thread and I can't seem to find the answer to why specifically the fwd (MON = N) is still restarted after being killer (simulating a crash).

Can you paste in the part I missed if I missed it or am not understanding?

Thanks,

Don

0 Kudos

Re: cpwd_admin list overview (SMS)

I think he meant:

Just to clarify, cpwd has two different mechanisms for monitoring processes.

Only specific processes can leverage the "active" mechanism (i.e. the MON column).

Arbitrary processes like the one you're trying to monitor for won't leverage it, but cpwd should still notice if the process is killed and restart it.

So some processes have Mon=N but are still restarted if they crash... You may find them in cpwd.elg.

Re: cpwd_admin list overview (SMS)

Just want to drop some extra info in here on RFL.

SmartEvent R80 Architecture to go with this description:

  • Manages the queries it get from the consumer processes, forward them to Solr and return the results. Also in charge of resolving and database maintenance (clean up old indexes to have space for the new ones).

R80SmartEventArchitecture.png

 

From:

https://www.checkpoint.com/downloads/products/r80.10-security-management-architecture-overview.pdf

We get this diagram:

LoggingDataFlow.PNG

 and this other info on debugging:

DEBUGGING Debug with Database Modifications. Use mgmt_cli.

In Pre-R80.10 systems, debugging was done using dbedit, which can still be used for the objects still managed from FWM (such as gateways, VSX objects and QoS policy).

Also in earlier systems, you could manual edit config files in $FWDIR/conf.

In R80.10, manual editing of files does not work. The configuration is stored in the PostgreSQL database, and not in the $FWDIR/conf files. R80.10 now uses these files to create a representation of the installed revision, on Install Policy.

R80.10 Logging Processes are from one CLI command set, rather than various tools of before.

SmartEventSetDebugLevel <component> <debug level>

Example:

RFL:

Start: SmartEventSetDebugLevel rfl debug

Stop: SmartEventSetDebugLevel rfl info

 

and finally this:

R8010-SMS-Processes.png

 

0 Kudos