Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Bruno_Ramos
Participant

fw zdebug drop fail to allocate memory

Jump to solution

Hi fellow CheckMates,

I am currently having a issue, a very strange one, with fw ctl zdebug, more accurately, when I issue "fw ctl zdebug drop" I get the following error.

kiss_kdebug_command: memory allocation failed, aborting.
Defaulting all kernel debugging options
Debug state was reset to default.

Does anyone ever saw this? If yes, how did you resolve it?

This is a VSX R80.30, and occurs in all member, fresh installed with the latest ISO (take200) and jhf 191.

Thank you and cheers

BR

 

0 Kudos
1 Solution

Accepted Solutions
Coby_Schmidt
Employee
Employee

Hi CheckMates,

 

The root cause of the issue is the exhaustion of the kdebug/zdebug address space which is limited to 4GB (this is a 32 bit process).

 

When using zdebug/kdebug the process will allocate memory in user-space for all the VSs and all the FW instances. The memory grows linearly with each FW instance in the system.

 

To work around it, use the following command to debug the specific VS.

 

fw ctl zdebug -v <vsid number> + drop

 

Please feel free to contact me @cobys@checkpoint.com any questions with this regards.  

View solution in original post

11 Replies
_Val_
Admin
Admin

zdebug is an internal macros that unfortunately leaked out and became way too popular for its purpose.

It is running debug commands with a buffer way too small for the purpose. More info here.

Why don't you run the debug in a proper manner:

fw ctl debug -buf 99999
fw ctl debug -m fw + drop
fw ctl kdebug -f > myfile.txt

when done, run

fw ctl debug 0

 

0 Kudos
HeikoAnkenbrand
Champion
Champion

Hi @Bruno_Ramos,

"fw ctl zdebug" is a powertool that is not exhausted from being used with "fw ctl zdebug drop". There is not much to be found in Check Point KB or in the documentation. "fw ctl zdebug" is an R&D tool for testing software in development. Therefore, the insert should be used with care. It starts a debugging in the background until it is aborted with CTRL+C. On productive systems it can have a high performance impact. Furthermore, the debug buffer is not the largest.

HeikoAnkenbrand_0-1589382606008.png

What happens when you execute! It is a macro that executes the following commands:

fw ctl debug -buf 1024           <<< The debug buffer is preset to 1024.
fw ctl debug [The option behind "fw ctl zdebug"]
fw ctl kdebug -f
       [Wait until CTRL+C is pressed]
fw ctl debug 0

More read in this article:
"fw ctl zdebug" Helpful Command Combinations

 

0 Kudos
Christian_Riede
Collaborator

Same here.

Works on 21800 Gaia R80.30 R155 VSX, fails with identical message on 23800 Gogo R80.30 T155 VSX.

Also, using fw ctl debug instead of zdebug does not solve it:

# vsenv 3
Context is set to Virtual Device ******* (ID 3).
# fw ctl debug -buf 99999
Initialized kernel debugging buffer to size 8192K
# fw ctl debug -m fw + drop
Updated kernel's debug variable for module fw
Debug flags updated.
# fw ctl kdebug -f > myfile.txt
# fw ctl debug 0
Defaulting all kernel debugging options
Debug state was reset to default.
# tail myfile.txt
Module: cluster
Enabled Kernel debugging options: None

-----------------------------------------------------
HOST:
Module: IDAPI
Enabled Kernel debugging options: None

-----------------------------------------------------
kiss_kdebug_command: memory allocation failed, aborting.

Problem with Gogo?

0 Kudos
_Val_
Admin
Admin

If you cannot start kernel debug with the maximum debug buffer, please open a support call with TAC

0 Kudos
Christian_Riede
Collaborator
6-0002003769 is already open.
0 Kudos
_Val_
Admin
Admin

Got it. Checking with R&D in parallel. 

0 Kudos
Bruno_Ramos
Participant

Hi all,

Thank you for all the replies.

I tried JHF 155 and 195 and the issue persist.

I have a SR open with TAC and R&D, as soon as i can I will update you all.

Thanks

BR

0 Kudos
Bruno_Ramos
Participant
Hi Christian,
I also tried JHF 195 and 155 on 21700 appliance but the issue persist.
Cheers
BR
0 Kudos
Christian_Riede
Collaborator
21700 would be gaia, not gogo? Here, on 21800 gaia R80.30 T155, it's working. The non working 23800 is gogo.
0 Kudos
Bruno_Ramos
Participant
Hi Christian,
using 21700 gaia.
Cheers
BR
0 Kudos
Coby_Schmidt
Employee
Employee

Hi CheckMates,

 

The root cause of the issue is the exhaustion of the kdebug/zdebug address space which is limited to 4GB (this is a 32 bit process).

 

When using zdebug/kdebug the process will allocate memory in user-space for all the VSs and all the FW instances. The memory grows linearly with each FW instance in the system.

 

To work around it, use the following command to debug the specific VS.

 

fw ctl zdebug -v <vsid number> + drop

 

Please feel free to contact me @cobys@checkpoint.com any questions with this regards.