Debug

[Don_Paterson]

Is there a plan to simplify the debug processes / procedures in the main train products or would it be a good idea to consider adding this? In other words, will there ever be a single command, for example 'debug' to use for debugging anything (and everything) in the Security Management and MDSM and Security Gateway products. Reason for asking is that it seems to be sometimes complicated to run a debug, which a customer may want to do in order to resolve issues themselves without getting support involved. Debugging fwd or fwm for example is fairly straightforward but a full debug of a policy installation has become much more complicated since R80. Thanks, Don

[Nick_Doropoulos]

@HeikoAnkenbrand has simplified the process of debugging policy installation on R80.x versions here:

https://community.checkpoint.com/t5/Policy-Management/R80-x-Debug-policy-installation-on-gateway/m-p...

Regarding a general debug, you can create a script consisting of several things to check which can then be run by the customer if needed.

*Please note that such a script however would place a tremendous load on the box's resources. If they warn of such consequences in the event of running a kernel debug you can appreciate what impact one could cause by debugging even more items at the same time.

[Don_Paterson]

Thanks. But I am not sure the fwm load <policy package name> <taget gw or cluster name> is still valid for a full policy install after R77.30 (so R80 and R80.x) - with or without -d for a onetime debug of the load/install task.

cpm_debug.sh may also be needed for a comprehensive debug of a policy compilation and installation.

Anyway, I am asking Check Point if they plan to in any way simplify the debug process in general.

I appreciate that they offer much more enhanced debug capabilities than competitors but if it can be made any easier then that would be a nice-to-have 'feature'. 

[G_W_Albrecht]

I would assume that CP tries to make debugs as simple as possible with a combination of very different products - if several different processes and daemons are each handling a part of the task chain it will get more complicated. To have one debug command fitting it all could be a valid RFE (you can suggest it here: https://rfe.checkpoint.com/rfe/rfe.htm), but it strongly reminds me of little childrens dreams of father Christmas 😉...

[Don_Paterson]

Ah, the old public RFE link 😀 😉

I appreciate what you have said and there are some interesting (good) things happening when doing debugs that you have to have a closer look to actually notice. An example is the ips debug (-o output.txt) command setting the fw kernel module flags vm drop spii cmi aspii advp ips in order to simplify the debug (I guess). Another example is setting flags on one kernel module can actually result in other kernel modules, but you have to run fw ctl debug to see those changes.

But then you wonder if that procedure will get forgotten about (not get updated) when updates to the software/product occur, which may not be the case if it was part of a global debug infrastructure (single root command or one repo). 

Maybe one script repo/director for all debug options (like we are seeing now in R80.x with the $FWDIR/scripts directory) for all processes would be good.

Of course fw ctl debug (zdebug) is always there/available but then we look at the user space processes and wonder why there are different commands for each daemon, apart from fw debug fwd and fw debug fwm of course...

[Shannon_Diotte]

I'm sure people would pay good money for a GUI app that allows you to step thru what you want to debug, then spit out the applicable commands. 😉

[Don_Paterson]

Sounds good 🙂

A bash script will do. Maybe it is already out there or in here. Always impressed with the information available on Check Mates and great to see the sharing going on.

Otherwise I always refer to memory or saved debug procedures, or do what TAC tells me when it is not obvious and is broken.

[Don_Paterson]

@PhoneBoy 

@_Val_ 

Hi guys,

I want to liven up this thread again, if only for an instant, and get an official reply from CP.

There have been good replies but I want to know if it (debug commands and procedures) is something that will change, since so many other changes are occurring in recent versions.

There is surely a valid argument to unify the debug process so that the majority of debugs can share a root command (debug for example).

I know enough to understand that the architecture of Check Point's software (very technical, and capable products) is probably going to make it possibly a bit complicated to satisfy that kind of request but when you look at the CPM debug procedure it seems more complicated than it needs to be.

That said, most customers do not need (or want) to debug their solution regularly. And I appreciate that too.

Regards,

Don

[PhoneBoy]

I'm not aware of any specific initiatives in this area, though I see where it would be useful.

To play devils advocate, though, there are also people who are used to doing the debug one way, and then you change it to something else, and they get upset.
Kinda like what happened when we tried to improve fw monitor syntax in R80.20 🙂

[Don_Paterson]

Thanks Dameon,

I understand what you are saying. But...

We are seeing things change so fast these days that we have to relearn many things. Check Point is not excluded.
That is not a bad thing because of all the changes in technology. That does bring more technology and so it seems to make the unified debug option more important.
There's that word unified 🙂

In R77 and older if I wanted to debug a policy install I would run:
fwm -d load PolicyName GatewayName or ClusterName
That was it.
Maybe an fw debug fwm on TDERROR_ALL_ALL=5 etc. but not much else.

This is the way it has been recommended (by Check Point) for me to do the policy install bug now.
There are at least a dozen steps:
Connect to the command line on the management server.
Backup the current $FWDIR/conf/tdlog.cpm file.
Edit the current file: vi $FWDIR/conf/tdlog.cpm
Change the logLevelInstalPolicy line to logLevelInstalPolicy=debug
Save the changes and exit the Vi editor.
Run commands to set and verify the required debug environment variable:
Export INTERNAL_POLICY_LOADING=1
ECHO $INTERNAL_POLICY_LOADING
Start the fwm debug: fw debug fwm on TDERROR_ALL_ALL=5
Use the appropriate syntax to verify the policy under debug.
Take all relevant outputs and screenshots.
Stop the fwm debug (fw debug fwm off).
unset INTERNAL_POLICY_LOADING environment variable (was set with =1).
Revert the original $FWDIR/conf/tdlog.cpm file.
cp –v $FWDIR/conf/tdlog.cpm $FWDIR/conf/tdlog.cpm_DEBUG
cp –vf $FWDIR/conf/tdlog.cpm_ORIGINAL $FWDIR/conf/tdlog.cpm
Send files to Check Point Support for analysis.

[Don_Paterson]

UPDATE:

Replying to my own reply here.

This is to acknowledge that Heiko has indeed posted the post (link pasted in below) regarding command line policy install in R80.x.

It is also mentioned above, in this thread. Thanks Nick 🙂

The procedure is 'one extra line' before the old version style command for CLI policy installation (fwm load)  and that will then also allow for us to run it in debug mode (fwm load -d) on the CLI.

The 'one extra line':

export INTERNAL_POLICY_LOADING=1

to be followed by (non-debug, which includes about a dozen lines of output):

fwm load <policy-name> <gateway/cluster-name>

The debug version (fwm load -d) gives us over 15,000 lines of output, which is great, but the question is - does that debug the whole procedure of policy install on the SMS?

For example, does that cover the CPM involvment (cpm debug) in the policy install (the DB dump part), in case that is relevant, without invoking the cpm_debug.sh debug script

I am sure TAC and R&D have procedures.

I am asking for clarity regarding the recommended debug procedure for policy installation (if an official recommendation is possible) and out of curiosity.

I can't give a use case. I just know that in the past and fwm load -d has given enough info to help out with diagnostics on site.

https://community.checkpoint.com/t5/Policy-Management/R80-x-Debug-policy-installation-on-gateway/m-p...

[G_W_Albrecht]

I would open an informational TAC case if this is very important to you 😎

[Don_Paterson]

😄
It's low priority 😉

[_Val_]

@Don_Paterson I am not sure which question you want Check Point to answer. Any chance we ask it again, please?

[Don_Paterson]

Hi Val,

How about this:

Will there ever be a single command, for example 'debug' to use for debugging anything (and everything) in the Security Management and MDSM and Security Gateway products?

Justification:

The product has increased in complexity and is changing fast.

A structured debug process would surely help.

More info:

Take for example the debug procedure above, as one example of a more complicated debug procedure compared to R77 (older versions)

Another example is if we consider the number of software architectural changes in recent versions and looking at SecureXL and the Kernel workings in general (UMFW and new fwaccel paths added) and how complex it might become to debug various aspects of those. Maybe that is not a good example but having a common debug procedure change with the software development and not the other way around seems to make sense.

Hope that helps.

Regards,

Don

[_Val_]

Not an official answer, but a personal opinion: very unlikely. 

I understand the growing complexity argument, and I think you need to adjust your approach . For myself, I have given up my thrive to debug R80.x management since it first appeared. It becomes way too heavy for deal with for a regular customer, and even for a trained engineer. 

Here is my argument:

1. On the field, one needs to troubleshoot, not to debug
2. The main purpose of troubleshooting is to understand where the problem is coming from
   1. If it is a config issue, one can probably fix it
   2. If not a config issue, support should fix it
3. In case troubleshooting is above one's skills, move the case to TAC

For TAC support engineers, mastering debugging tools is much easier for mane reasons: they do it on a daily basis, they are very close to the source code, they have internal know-how and documentation. 

For customers, I do not think debugging skills should be a priority. Maintaining those skills require practice, lots of training and hands-on time, and also poses lots of risks, if done in production.

I have been teaching troubleshooting courses for a decade, and my firm belief is that customers do not need 90% of debugging techniques they are exposed to.

Most importantly, from liability and risk management perspective, it is much better to let your support partner or TAC engineer to  debug than doing it yourself. If something goes wrong... 🙂

One need a good understanding of architecture and dependancies to troubleshoot. Java debug level 6 is too much to ask from a network engineer, let TAC and developers to give you a hand here.

I understand this is not an answer you would like to get. 

[Don_Paterson]

Thanks Val, It is a good answer and I do like it, mostly 🙂
You make valid points that are true.
Appreciate the time you've taken to reply with your thoughts.

Knowledge of the system is indeed critical for troubleshooting and making diagnosis.
Check Point has done great things to help, for example cpview, some $FWDIR/scripts stuff, and the health check scripts and DiagnosticsView.
These tools and the debug procedures (when applicable) have to be learned and used properly. By that I mean that they should be known about and used with the right switches, like -T for timestamps and -d (or -D) for a once off debug.
I still think some troubleshooting could be made easier with a global 'debug' command. 🙂

Regards,
Don

[_Val_]

Fair enough, @Don_Paterson. 

From a development point of view, we are talking about a mixture of different codes, software layers and programming languages. Unification of debug commands would require a huge all-through R&D project, and as I tried to explain, this effort is too much it terms of actual field need.

I would rather have my developers producing new secure features 🙂

[Don_Paterson]

Well well, look at that. More new daemons and unique debug commands to go with it 🙂

https://sc1.checkpoint.com/documents/R81.10/WebAdminGuides/EN/CP_R81.10_CLI_ReferenceGuide/Topics-CL...

ike debug

Background

Starting in R81.10, separate daemons handle different VPN connections:

• The VPN daemon vpnd.

  Handles these VPN connections:

  • Site-to-Site connections from peer Security Gateways with a Statically Assigned IP address

  • All connections from non-IPsec Remote Access clients (SSL Network Extender)

  • Multi-Portal traffic

  Listens on these ports on a Security Gateway

  

  :

  • IKE: 500 (UDP)

  • IKE NAT-T: 4500 (UDP)

  • TCPT: 444 (TCP)

  • Session infrastructure manager: 9996 (TCP)

  This process is a child of the FWD process (see the $FWDIR/conf/fwauthd.conf file on a Security Gateway).

• The IKE daemon iked (introduced in R81.10).

  Handles these VPN connections:

  • All connections from IKE Remote Access clients clients (for example, Endpoint clients)

  • Site-to-Site connections from peer Security Gateways with a Dynamically Assigned IP address (DAIP)

  • Large Scale VPN (LSV) connections

  • Connections from SmartLSM ROBO gateways

  Listens on these ports on a Security Gateway:

  • IKE: 30500 (UDP)

  • IKE NAT-T: 34500 (UDP)

  • Session infrastructure manager: 9994 (TCP)

  • L2TP: 1701 (UDP)

  Syntax: vpn iked

  This process is a child of the FWD process (see the $FWDIR/conf/fwauthd.conf file on a Security Gateway).

• The CCC daemon cccd (introduced in R81.10).

  Responsible for the Circuit Cross-Connect (CCC) protocol, while:

  • IKE for the same clients runs in the IKE daemon iked

  • CCC TLS for the same clients runs in the VPN daemon vpnd

  Listens on these ports on a Security Gateway:

  • TCPT: 444 (TCP)

  • Session infrastructure manager: 9993 (TCP)

  Syntax: vpn cccd

  This process is a child of the FWD process (see the $FWDIR/conf/fwauthd.conf file on a Security Gateway).

Disabling the IKE daemon "iked"

It is possible to configure the Security Gateway to disable the IKE daemon iked and work in the legacy mode, in which the VPN daemon vpnd handles all VPN connections.

Syntax

ike debug

      on [<Debug_Topic>=<Debug_Level>]

      off

      ikeon [-s <Size_in_MB>]

      ikeoff

      trunc [<Debug_Topic>=<Debug_Level>]

      truncon [<Debug_Topic>=<Debug_Level>]

      truncoff

      timeon [<Seconds>]

      timeoff

      ikefail [-s <Size_in_MB>]

      mon

      moff

      say ["String"]

      tunnel [<Level>]

[_Val_]

Attention, quoting from Important security update - stay protected against VPN Information Disclosure (CVE-2024-24919)

In R81.10 we added a feature to improve VPN performance - named CCCD

This feature is disabled by default, and we know about few advanced customers who are using it.

Customers who enable CCCD are still vulnerable to CVE-2024-24919 even after installing the Hotfix!

YOU MUST DISABLE CCCD TO BECOME PROTECTED!

Instructions below and also on SK182336:

Run the command: vpn cccd status
The expected output is: vpn: 'cccd' is disabled.

If the output differs, stop the CCCD process by running the vpn cccd disable command.

More info by the link above.