cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post

How to check debug command ?

Jump to solution

Hi Everyone,

I have been challenging from customer, they would like to know if there is any commands to check whether any debugging command is running at a time so that they will able to stop those command right away in case of some admins system or TAC forget to turn it off after running debug.

As my understanding from Top commands would be possible but not quite sure.

Really appreciate every comments

Regards,

Sarm

1 Solution

Accepted Solutions

Re: How to check debug command ?

Jump to solution

In general there are two primary areas of debugging: Process Space and Kernel Space.  An SMS will only have debugs available in Process Space, while a gateway can have debugging active in Kernel Space and/or Process Space.  Kernel Space debugs are far more likely to cause gateway performance or stability effects if they get into a runaway state.

For kernel debugs you can see what debug flags are currently set with these commands: 

fw ctl debug

sim dbg list (R80.20+ - fwaccel dbg list)

To reset kernel debugs to default:

fw ctl debug 0

sim dbg resetall (R80.20+ - fwaccel dbg resetall)

Note: for SecureXL debugs (sim/fwaccel) it is extremely important to set a very specific filter with the -f option or the chances of cratering the system with a runaway debug are very high.

Because there are so many different tools and techniques for initiating Process Space debugs, figuring out if one is active is much more difficult.  Probably the best approach would be to run these commands and look for *.elg files rapidly increasing in size where the process debug files are typically written, and whether they are quickly being rotated (i.e. fwd.elg, fwd.elg.1, fwd.elg.2):

watch ls -ltr $FWDIR/log/*.elg

watch ls -ltr $CPDIR/log/*.elg

--
"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
9 Replies

Re: How to check debug command ?

Jump to solution

Hi Günther W. Albrecht,

Thanks for sharing, but I would like to get the commands that can display lists of debugging is running at that time.

Regards,

Sarm

0 Kudos

Re: How to check debug command ?

Jump to solution

Sorry, but i find that this is going nowhere fast ! Every Debug has to be planned in detail and scheduled, and every production gateway / SMS needs a maintenance window for debugs (maybe excluding e.g. policy install debugs). If you can imagine a situation with multiple administrators connected to the same device and one is debugging VPN and second one debugs CoreXL, you are not in the security business but purely into show business 😉

Petr_Hantak
Silver

Re: How to check debug command ?

Jump to solution

You are right about service window no doubt. You should have it always for debugs. If you have service window, then you should notice anyone else about it, steps should be reviewed annd correct and so on. But what if somebody ends debug incorrectly no matter on reason? For example 

[Expert@HostName]# vpn debug trunc
[Expert@HostName]# vpn debug on TDERROR_ALL_ALL=5

runs on background until you turn it off properly.

I know it is pure theory but it could be possible. 

Chanatip Adisaktrakool‌ I think you should try to explain customer that situation shouldn't occur. I have experience that TAC is very careful with debugs and they are alway ending it and it is responsibility of administrator to run and end debug properly in agreed service window like Günther W. Albrecht‌ wrotes above.

0 Kudos

Re: How to check debug command ?

Jump to solution

I know that this can occur - but the question is about a command showing any debugs currently configured / running, and there is no such command. Except top, as a daemon under debug will need much more ressources 😉

0 Kudos
Petr_Hantak
Silver

Re: How to check debug command ?

Jump to solution

Yeah even it is tricky that we have so many debug types, it is still relevant question. I can imagine situation when you have multiple administrators connected to the same device and one is debugging VPN for example and second one reacting to some monitoring event for CPU and wants to debug CoreXL for example. In case both runs debugs in the same time, they could easilly kill the device just because they don't know about each other. 

0 Kudos

Re: How to check debug command ?

Jump to solution

Hi Petr,

Yeah, you're right Smiley Happy

Regards,

Sarm

0 Kudos

Re: How to check debug command ?

Jump to solution

In general there are two primary areas of debugging: Process Space and Kernel Space.  An SMS will only have debugs available in Process Space, while a gateway can have debugging active in Kernel Space and/or Process Space.  Kernel Space debugs are far more likely to cause gateway performance or stability effects if they get into a runaway state.

For kernel debugs you can see what debug flags are currently set with these commands: 

fw ctl debug

sim dbg list (R80.20+ - fwaccel dbg list)

To reset kernel debugs to default:

fw ctl debug 0

sim dbg resetall (R80.20+ - fwaccel dbg resetall)

Note: for SecureXL debugs (sim/fwaccel) it is extremely important to set a very specific filter with the -f option or the chances of cratering the system with a runaway debug are very high.

Because there are so many different tools and techniques for initiating Process Space debugs, figuring out if one is active is much more difficult.  Probably the best approach would be to run these commands and look for *.elg files rapidly increasing in size where the process debug files are typically written, and whether they are quickly being rotated (i.e. fwd.elg, fwd.elg.1, fwd.elg.2):

watch ls -ltr $FWDIR/log/*.elg

watch ls -ltr $CPDIR/log/*.elg

--
"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
Highlighted
Bryce_Myers
Nickel

Re: How to check debug command ?

Jump to solution

I would recommend downloading the healthcheck script from sk121447 and looking at the section called "check_debugs()".

Or just run the healthcheck script on the gateway and read the output about the debug configurations.