topic Re: fw zdebug drop fail to allocate memory in Firewall and Security Management

fw zdebug drop fail to allocate memory

Bruno_Ramos — Wed, 13 May 2020 13:44:41 GMT

Hi fellow CheckMates,

I am currently having a issue, a very strange one, with fw ctl zdebug, more accurately, when I issue "fw ctl zdebug drop" I get the following error.

kiss_kdebug_command: memory allocation failed, aborting.
Defaulting all kernel debugging options
Debug state was reset to default.

Does anyone ever saw this? If yes, how did you resolve it?

This is a VSX R80.30, and occurs in all member, fresh installed with the latest ISO (take200) and jhf 191.

Thank you and cheers

Re: fw zdebug drop fail to allocate memory

_Val_ — Wed, 13 May 2020 14:42:45 GMT

zdebug is an internal macros that unfortunately leaked out and became way too popular for its purpose.

It is running debug commands with a buffer way too small for the purpose. More info here.

Why don't you run the debug in a proper manner:

fw ctl debug -buf 99999
fw ctl debug -m fw + drop
fw ctl kdebug -f > myfile.txt

when done, run

fw ctl debug 0

Re: fw zdebug drop fail to allocate memory

HeikoAnkenbrand — Wed, 13 May 2020 15:14:08 GMT

Hi @Bruno_Ramos,

"fw ctl zdebug" is a powertool that is not exhausted from being used with "fw ctl zdebug drop". There is not much to be found in Check Point KB or in the documentation. "fw ctl zdebug" is an R&D tool for testing software in development. Therefore, the insert should be used with care. It starts a debugging in the background until it is aborted with CTRL+C. On productive systems it can have a high performance impact. Furthermore, the debug buffer is not the largest.

What happens when you execute! It is a macro that executes the following commands:

fw ctl debug -buf 1024 <<< The debug buffer is preset to 1024.
fw ctl debug [The option behind "fw ctl zdebug"]
fw ctl kdebug -f
[Wait until CTRL+C is pressed]
fw ctl debug 0

More read in this article:
"fw ctl zdebug" Helpful Command Combinations

Re: fw zdebug drop fail to allocate memory

Christian_Riede — Mon, 18 May 2020 06:17:47 GMT

Same here.

Works on 21800 Gaia R80.30 R155 VSX, fails with identical message on 23800 Gogo R80.30 T155 VSX.

Also, using fw ctl debug instead of zdebug does not solve it:

# vsenv 3
Context is set to Virtual Device ******* (ID 3).
# fw ctl debug -buf 99999
Initialized kernel debugging buffer to size 8192K
# fw ctl debug -m fw + drop
Updated kernel's debug variable for module fw
Debug flags updated.
# fw ctl kdebug -f > myfile.txt
# fw ctl debug 0
Defaulting all kernel debugging options
Debug state was reset to default.
# tail myfile.txt
Module: cluster
Enabled Kernel debugging options: None

-----------------------------------------------------
HOST:
Module: IDAPI
Enabled Kernel debugging options: None

-----------------------------------------------------
kiss_kdebug_command: memory allocation failed, aborting.

Problem with Gogo?

Re: fw zdebug drop fail to allocate memory

_Val_ — Mon, 18 May 2020 07:14:55 GMT

If you cannot start kernel debug with the maximum debug buffer, please open a support call with TAC

Re: fw zdebug drop fail to allocate memory

Christian_Riede — Mon, 18 May 2020 07:17:15 GMT

6-0002003769 is already open.

Re: fw zdebug drop fail to allocate memory

_Val_ — Mon, 18 May 2020 07:46:23 GMT

Got it. Checking with R&D in parallel.

Re: fw zdebug drop fail to allocate memory

Bruno_Ramos — Mon, 18 May 2020 15:42:25 GMT

Hi all,

Thank you for all the replies.

I tried JHF 155 and 195 and the issue persist.

I have a SR open with TAC and R&D, as soon as i can I will update you all.

Thanks

Re: fw zdebug drop fail to allocate memory

Bruno_Ramos — Mon, 18 May 2020 15:43:30 GMT

Hi Christian,
I also tried JHF 195 and 155 on 21700 appliance but the issue persist.
Cheers
BR

Re: fw zdebug drop fail to allocate memory

Christian_Riede — Mon, 18 May 2020 15:52:15 GMT

21700 would be gaia, not gogo? Here, on 21800 gaia R80.30 T155, it's working. The non working 23800 is gogo.

Re: fw zdebug drop fail to allocate memory

Bruno_Ramos — Mon, 18 May 2020 15:57:05 GMT

Hi Christian,
using 21700 gaia.
Cheers
BR

Re: fw zdebug drop fail to allocate memory

Coby_Schmidt — Wed, 20 May 2020 19:34:18 GMT

Hi CheckMates,

The root cause of the issue is the exhaustion of the kdebug/zdebug address space which is limited to 4GB (this is a 32 bit process).

When using zdebug/kdebug the process will allocate memory in user-space for all the VSs and all the FW instances. The memory grows linearly with each FW instance in the system.

To work around it, use the following command to debug the specific VS.

fw ctl zdebug -v <vsid number> + drop

Please feel free to contact me @cobys@checkpoint.com any questions with this regards.