fw ctl zdebug - yea or nay?

[_Val_]

Hello everyone. 

My position towards zdebug practice is known for quite some time. On every Troubelshooting course relivede I was explaining why I do not want my students to use fw ctl zdebug. Last year I have also published an article at my blog about this: CCMA's blog: Kernel debug Best Practices or "Why "fw ctl zdebug..." should not be used" 

However, since this undocumented macros leaked out from Check Point R&D about 15 years ago, I see more and more people using it, and to my dismay, SK articles with this command. 

So, to put the matter to rest, I want to check what you guys are thinking about this practice. Let's put it to the vote. The poll will be closing in two weeks from now.

Thanks for participating. 

I use it all the time, with different flags and all. I like it	22
I only use "fw ctl zdebug drop" when I am on the hurry to fix a burning issue. I understand its limitations and agree it is to avoid	31
I never use it and I discourage others from using it.	3

[HeikoAnkenbrand]

I think there‘s a simple answer:

In lab environments - Yes

In production environments - No

I think it is a R&D tool for quick debugging.

You can found more infos in the following article:

"fw ctl zdebug" Helpful Command Combinations 

PS: Valeri, there is a small typing issue „zdebud“ in the headline.

Regards,

Heiko

[PhoneBoy]

I fixed the typo at least

[HeikoAnkenbrand]

Regards

Heiko

[_Val_]

thanks, missed that

[_Val_]

you realized nobody reads the warning and gets straight into action after reading your piece? 🙂

[_Val_]

Guess what, found another typo. Fixed also

[Yuri_Slobodyany]

I agree partly with  Heiko Ankenbrand‌:

- In LAB environment - yes
- In production - yes

- As a SmartView Tracker replacement drop-in   (to see if a rule drops the packet, who has time to open SmartConsole?) - yes

- Antispoofing/routing misconfiguration check - yes

[Hugo_vd_Kooij]

I use it slightly more then just with the drop option. But only if other means are not conclusive.

After all it IS a debug command and debugging takes up resources.

[_Val_]

How about the buffer size?

[HeikoAnkenbrand]

It is a macro that executes the following commands:

fw ctl debug -buf 1024
fw ctl debug [The option behind "fw ctl zdebug"]
fw ctl kdebug -f
       [Wait until CTRL+C is pressed]
fw ctl debug 0

So the buffer is not very large:-) I think that could be a problem with "fw ctl zdebug drop", too. Therefore, from my point of view, it's just for the lab environment.

Regards

Heiko

[Jerry]

+ drop with few specs afterwards but never in PROD 😉 Agree with all of you here all above. Awesome tool!

[_Val_]

Hi, I assume Hugo knows that, hence the question 🙂

[Maarten_Sjouw]

Fully agree, just use it when needed, most of the times when Tracker/Smartlog does not show the traffic being dropped zdebug drop will.

[_Val_]

Any proper debug will show you that. Are you familiar with fw ctl debug tool?

[Petr_Hantak]

I remember the article from Valeri Loukine‌ very well. Basically I agree with idea do not use it or only in limited environment, but still it is practical so:

• In lab - yes I use it in more variations
• In producition - yes I use it only "fw ctl zdebug drop" to quickly check dropped or antispoofing issued connections

Why we shows "fw ctl zdebug drop" to our colleagues on internal training sessions? Because it is easy to use. Especially when you have on-call specialist who should usually cover very large portfolio of different platforms in whole network area nowadays, you should give them quick&easy opportunity to check the issue.

[_Val_]

Okay, here is my issue, right there. There is nothing quick&easy about kernel debug. That seems easiness is the main issue. Whoever does kernel debug, needs to understand the tool, the process and the implication. When encouraging people to use fw ctl zdebug, you show them a shortcut without explaining the mechanism.

[Petr_Hantak]

Yes with no doubt speaking about debug on Check Point it is not easy at all. You need understand so many things. What we are trying to do is try to explain the way to specialists, share background with them, warn them about limitation and impact. That is everything nice but if you are not working with Check Point every day and this is even not expect from everyone on the department, they remember less there 10% of knowledge that you shared. From on-call is expected to be superhero who is able to fix or start troubleshoot everything until TAC or relevant master guy is available. Plus when they are already wants to start debugs, everything is already in the fire. So still we share info that they can use it with risks and limitations and in case you are not sure what do you want or you are expect impact then "DO NOT USE IT!" and wait for relevant person.

It is not optimal for sure and possibly dangerous but during burning issue you have usually no time.

[Yuri_Slobodyany]

Actually a good point https://community.checkpoint.com/people/petr.1a4c7ef0-1799-4f26-8940-31a78579d64e - it greatly depends on the perspective, working for a Checkpoint Partner and taking care of hundreds of CP firewalls I forget that most IT folks work in Enterprises and need to debug once in a while one or few firewall boxes they have. In such cases playing with debug not fully understanding what you see can be indeed counterproductive - you can see 'problems' that do not exist or miss the ones that are actually there.

[_Val_]

Petr, I used to run my troubleshooting course for the last 10 years. At the beginning of the course I always say: Check Point is simple, and we have three days to prove it. I stand by this even now. All you need is structure and understanding of architecture. Then it all falls into place. 

It is better to explain how things are working than giving them hacks. fw ctl zdebug is one of those hacks. 

[Jerry]

I love your comment buddy it makes so much sense  very well said! kudos Valeri Loukine‌

[Hugo_vd_Kooij]

For anything more complicated I usually figure out my own script.

Preferably with 2 extra SSH sessions active.

 1. runs top on a 1 second interval

 2. only requires me to hit enter. It allready has fw ctl debug 0 typed in. (don't forget the timeout on the console!!!)

These days I can guess a bit how much impact certain debugging flags will have. Some of them will send a box straight to full system overload. Even if the system is using only 5% CPU before the debug.

[_Val_]

>>> These days I can guess a bit how much impact certain debugging flags will have. Some of them will send a box straight to full system overload. Even if the system is using only 5% CPU before the debug.

Exactly

[Jeroen_Demets]

Hi,what if someone previously ran debug and forgot to stop debugging?

Actually about all debugging commands...do they survive reboot? I hope not because a reboot is the most easy way to stop all those forgotten started debug commands.

If not, I'll start using fw ctl debug 0 too (next to unset TMOUT) when logging in next time.

[_Val_]

kernel debug will not survive reboot.

[_Val_]

Yuri Slobodyanyuk‌ Petr Hantak‌ 

You may want to look into this: https://community.checkpoint.com/message/26028-tool-httpstcpdump101com  It is might be much better alternative than fw ctl zdebug

[Petr_Hantak]

Nicely said  Well of course we are sharing background and tips in our course with my colleagues as well. We want to transfer as much useful knowledge as possible but we must count that only some minor piece of all will be remembered. 

Even we have almost 100 specialist only couple of them we have fully focused on Check Point. Rest of them covers other platforms or need to do just simple things across more platforms.

In result you can't expect that 8 months after course they will be able to remember all background which you gave them. They uses written presentations or documentation of course. Even that they open TAC on shift in case of Major incidents it is expected that you'll act before you'll get engineer on-line. So that is the reason why we want to make their life easier.

Thanks to this community we can improve it for sure. Topic likes Common Check Point Commands (ccc)‌ and other wonderful scripts helps us a lot.

In conclusion for the future it is not a bad idea just "forget" to present zdebug and remove it from materials.

[Petr_Hantak]

Thanks Valeri to point me there. I really love this one!!!

[_Val_]

I wish it were that simple. Too many smart people in R&D, hazard of the trade. They will come up with better hacks, trust me 🙂 Keeping those in house is a challenge