topic Re: Troubleshooting dropped packets in Checkpoint using zdebug in Firewall and Security Management

Troubleshooting dropped packets in Checkpoint using zdebug

Zaid_Khan — Thu, 05 Jan 2023 15:21:36 GMT

Ever wished you had more insight into the traffic getting dropped by your Checkpoint Firewall?

Read on to learn a very powerful tool you to your rescue known as zdebug.

The fw ctl zdebug drop command lists all dropped packets in real time and explains the reasons for the drop

Use the expert mode fw ctl zdebug drop CLI command to set all the debugs flags and get an output on the command line.

The syntax for the command is:

[Expert@hostname]# fw ctl zdebug + <flags>

where <flags> could be any fw module flag.

For Example: The most common usage is the drop command:

[Expert@hostname]# fw ctl zdebug + drop

If you want to see drops only for a single IP use the grep filter:

[Expert@hostname]# fw ctl zdebug + drop | grep X.X.X.X

Replace X.X.X.X with the IP you want to filter for.

If you still cannot see the traffic, then most likely traffic is not even hitting the firewall. To verify, you can use tcmdump utility to capture packets:

Open a new session and:

[Expert@hostname]# tcpdump -nni any host x.x.x.x -s0 -w /var/log/tcpdump1.pcap

Note: The zdebug starts a debug in the background until it is stopped using CTRL + C.

Note: When I did CTRL +C to stop the captures, I got the following notification:

Next time perform for exit: "fw ctl debug 0"

Cannot unset debug filter

So you might need to do this as well to completely stop all debugs:

[Expert@hostname]# fw ctl debug 0

Defaulting all kernel debugging options

Debug state was reset to default.

PPAK 0: Get before set operation succeeded of simple_debug_filter_off

Re: Troubleshooting dropped packets in Checkpoint using zdebug

the_rock — Thu, 05 Jan 2023 16:07:09 GMT

Definitely good post for any troubleshooting.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

Chris_Atkinson — Thu, 05 Jan 2023 16:34:54 GMT

sk100808 also describes this and I know the topic is a trigger for @_Val_

Please also consider the alternatives to tcpdump being... fw monitor & cppcap

Re: Troubleshooting dropped packets in Checkpoint using zdebug

_Val_ — Thu, 05 Jan 2023 17:47:21 GMT

Since you tagged me here, @Chris_Atkinson...

There is no need to run "fw ctl debug 0" just because this command is done by zdebug macros when you press ctrl-z on exit.

The rest... It is sad and disappointing that more and more people are using zdebug instead of a proper kernel debug, but I lost my battle after it leaked to SecureKnowledge and was widely adopted by TAC engineers. I would expect someone at R&D actually to open the code and at least add a decent debug buffer to it instead of 1024K as it is now. But no, nobody listens. And now we have long posts about that crap.

All I had to say is said here: http://checkpoint-master-architect.blogspot.com/2017/11/kernel-debug-best-practices-or-why-fw.html

I rest my hopeless case...

Re: Troubleshooting dropped packets in Checkpoint using zdebug

the_rock — Thu, 05 Jan 2023 18:03:16 GMT

Good sk for sure. Well, speaking of zdebug and kernel debug, I will share 2 quick stories that come to my mind about it. So once, I was working with this large bank and they had case with escalations and there was guy from R&D on a call who had kernel debug prepared and he assured the customer all was fine and would you know it, as luck would have it, as soon as he ran the debug, fw got stuck, no one could access it and poor client had to drive hour away to console into it and get it working again. Lets just say, they were NOT happy (to put it nicely).

Then, 2nd time, different customer and I were trying to fix weird issue they had after upgrade to R80.30 and since Tier 2 and Tier 3 could not help, they transferred us to escalations and guy asked us to run kernel debug and sure enough, box got stuck again and we had to physically powercycle it. Thank God device was 20 feet away and not 100 miles (phew).

So, in all honesty, I cant blame TAC for sticking with zdebug, I do the same all the time. I never ever recommend any customers to do kernel debug, even if TAC suggests it, unless they have super solid reason for it.

Just my opinion based on bad experiences with it in the past.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

_Val_ — Fri, 06 Jan 2023 08:40:32 GMT

@the_rock Lol, did you just seriously write all this? What do you think is the difference between "fw ctl debug" process and "fw ctl zdebug"?

Re: Troubleshooting dropped packets in Checkpoint using zdebug

G_W_Albrecht — Fri, 06 Jan 2023 09:10:30 GMT

Not any news here - why did you post this very old information ? Kernel debugs with a small buffer like here remember me of R65 debugs...

Re: Troubleshooting dropped packets in Checkpoint using zdebug

the_rock — Fri, 06 Jan 2023 10:04:23 GMT

@_Val_ lol, I did seriously write all that. Maybe if there was proper kernel debug out there, more people would be running it, instead of zdebug.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

G_W_Albrecht — Fri, 06 Jan 2023 10:23:01 GMT

Please explain the difference between "fw ctl debug" process and "fw ctl zdebug"! Afaik, fw ctl zdebug runs a "fw ctl debug" kernel debug. It is only a shorthand way of defaulting all kernel parameters, setting the buffer to 1MB, and then adding fw module flags. ("fw ctl debug")

Issue here is that the 1MB buffer is so small that it is very restrictive in use. Manual "fw ctl debug" gives you much more parameters to fine tune debug and make it less hard for the GW.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

Tobias_Moritz — Fri, 06 Jan 2023 10:22:48 GMT

@the_rock : I also saw locked-up gateways due to kernel debugs suggested by TAC, but this has nothing to do with usage of zdebug macro or not. You are so long here, did you ever read the blog post from Val from 2017 (it is even linked here)? He clearly says that zdebug macro is nothing more like this:

fw ctl debug -buf 1024
fw ctl debug (your options)
fw ctl kdebug -f
-------(waiting for Ctrl-C)
fw ctl debug 0

Because of this and the way to small buffer, it should be avoided and is bad practice. I argue everytime with TAC, when they suggest using it. Beside other deprecated things like tcpdump (instead of cppcap), fw monitor -f (instead of -F) and other things.

I can only second what Val (and Günther) wrote. This way of troubleshooting is outdated and should not be used.

When you did not saw crashes with "fw ctl zdebug drop" but with some "fw ctl debug" procedure, this is not because the use of the zdebug macro, but because of the debug options (drop versus other options TAC gave your for the specific support case) or the specific environment.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

G_W_Albrecht — Fri, 06 Jan 2023 10:29:19 GMT

Debug on a productive environment without maintenance window ? No one of the customers i know would let anyone do that ! Also i do not see the point of telling such tales for zdebug - looks like you assume that the two GW downs would not have been happening if zdebug had been used instead of fw ctl debug. But this in only your - unprooven - thought, not the truth...

Re: Troubleshooting dropped packets in Checkpoint using zdebug

the_rock — Fri, 06 Jan 2023 10:58:01 GMT

Im so glad we are having this discussion @G_W_Albrecht . To answer your post, no, it was always done in maintenance window, NOT working hours and 2 stories I told, they are just 2 that stick in my mind, but there at least dozens more. And yes, I am confident had zdebug been used, this would have never happened, because many times, with TAC on the phone, we did zdebug on boxes running 99% CPU and never an issue, so if thats not good enough proof, then not sure what is.

@Tobias_Moritz Yes, I read @_Val_ link. I think it does bring up some good points, but again, if you guys believe those flags are outdated and should not be used, maybe everyone at CP should get on the same page and publish an official sk about it, so there is no confusion.

All I can say is this...and its 100% honest feedback. I had seen other TAC engineers from major vendors (PAN, Fortinet, Cisco) do debugs on their firewalls in production hours all the time and not a single crash.fw lock up.

I cant even count how many times when I talked to TAC and they mentioned kernel debugs, I would tell them one of those 2 stories from my last post and answer would be sort of "Well, I hear you, but escalations/R&D may need those debugs down the road". That does not inspire confidence to the customer if it will cause their firewall to crash...so, even TAC engineers know its bad.

Again, Im not trying to diminish the steps, all Im saying is I wish there were better steps to collect those debugs, thats all.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

_Val_ — Fri, 06 Jan 2023 11:16:45 GMT

Yes sir, @G_W_Albrecht, and if @the_rock was a bit more attentive, he could actually also discover that fact, instead of making ridiculous suggestions, just by reading my link referenced above, where I explain that.

Andy, please stop making these claims, it is no longer amusing 🙂

Re: Troubleshooting dropped packets in Checkpoint using zdebug

_Val_ — Fri, 06 Jan 2023 11:32:30 GMT

@the_rock Oh boy, I do not know what else I can tell you, other than beg you to re-read what is already said here.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

_Val_ — Fri, 06 Jan 2023 11:35:42 GMT

@Tobias_Moritz I mentioned the link above 🙂

Re: Troubleshooting dropped packets in Checkpoint using zdebug

the_rock — Fri, 06 Jan 2023 12:04:15 GMT

@_Val_ Im not being amusive, Im being 100% serious. Speak to TAC engineers yourself and get a feedback from them. If firewalls getting locked up and not having access is ridiculous, then I truly have nothing else to say. I rest my case.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

_Val_ — Fri, 06 Jan 2023 12:08:18 GMT

🤦

Re: Troubleshooting dropped packets in Checkpoint using zdebug

the_rock — Fri, 06 Jan 2023 12:16:15 GMT

@_Val_ ...lets have civil discussion. Please tell me what Im missing here.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

_Val_ — Fri, 06 Jan 2023 13:00:37 GMT

You are missing a very simple fact: zdebug is in fact nothing but a set of "fw ctl debug" commands, wrapped into a macros.

You keep claiming there is a fundamental difference between using "fw ctl debug" and "fw ctl zdebug", while the main difference between these techniques is that zdebug reserves a very small output buffer, which makes it very impractical when used in a production environment.

This fact is thoroughly described in my blog post linked above, and then re-told to you by two other fellow community members. now I repeat it again, the fourth time.

You claim zdebug is somehow safer than the regular kernel debug, which is not true. You refer to an occurrence when kernel debug crashed a FW. You somehow convinced yourself that it would work better with zdebug, and once again, you are mistaken. I have seen multiple cases in my 25 years where zdebug had exactly the same effect, but referring to personal experience is pointless. Kernel crash is just bad luck plus kernel code bugs, and actually, with a smaller debug buffer (zdebug) the chances of a crash are somewhat higher than with a bigger buffer.

I kindly ask you to re-read carefully the thread and my blog post, and if you have any questions about the matter, I will be happy to answer them.

However, if you still do not see why your statements were incorrect, I am afraid I have to give up and close this discussion.

Re: Troubleshooting dropped packets in Checkpoint using zdebug

the_rock — Fri, 06 Jan 2023 13:49:09 GMT

THANK YOU @_Val_ . Now that I read your response and also your blog more carefully, its definitely way more clear to me. Having said that, I will add that its not just my own experience about kernel debug, but I had many customers tell me the same. Anyway, everyone would have had different "journey" when it comes to this...

I do have a suggestion though. Do you think it would be possible to maybe chase R&D and put an official public SK together that would outline proper debugs/commands to use? I believe it would be way better to have that on support site than people "venting" here about it...just my opinion 🙂