Unusual zdebug Output

[RemoteUser]

Hi everyone,
I hope you're all doing well.

The client recently reported a rather unusual error that I have to admit I've never seen before. After running a zdebug drop on the firewall to monitor dropped packets, the following output was observed:

Has anyone come across something similar?

I'm wondering if this behavior could be related to the fact that SecureXL is running in UPPAK mode. Could this debugging method possibly interfere or produce unexpected results in such a configuration?

fw ctl zdebug + drop | grep x.x.x.x
^C
Next time perform for exit: "fw ctl debug 0"

cpdev_user_ioctl: ioctl failed to device /vs0/cp/usim/main
: Interrupted system call
*** Error in `fw': corrupted size vs. prev_size: 0x0879db68 ***
======= Backtrace: =========
/lib/libc.so.6(+0x78eaf)[0xefbaaeaf]
/lib/libc.so.6(__libc_calloc+0xb3)[0xefbad113]
/opt/CPshrd-R81.20/lib/libOS.so(+0x464c6)[0xf06344c6]
/opt/CPshrd-R81.20/lib/libOS.so(+0x46a07)[0xf0634a07]
/opt/CPshrd-R81.20/lib/libOS.so(CpSystemEx+0x161)[0xf0634f31]
/opt/CPshrd-R81.20/lib/libOS.so(CpSystemWithFlags+0x2b)[0xf063513b]
/opt/CPshrd-R81.20/lib/libkiss_apps.so(kiss_debug_command+0x14f7)[0xf1649eb7]
/opt/CPshrd-R81.20/lib/libkiss_apps.so(kiss_zdebug_command+0x426)[0xf164e046]
/opt/CPsuite-R81.20/fw1/lib/libfw1.so(+0x118f3f)[0xf38b3f3f]
/opt/CPsuite-R81.20/fw1/lib/libfw1.so(main_fwctl+0x4b4)[0xf38b7674]
/opt/CPsuite-R81.20/fw1/lib/libfw1.so(fw_cmain+0xccf)[0xf38fca2f]
fw(main+0xa8)[0x8052b48]
/lib/libc.so.6(__libc_start_main+0xde)[0xefb4a3ae]
fw[0x8052be5]
======= Memory map: ========
08048000-08086000 r-xp 00000000 fd:01 17622430                           /opt/CPsuite-R81.20/fw1/bin/fw_full
08086000-08087000 r--p 0003d000 fd:01 17622430                           /opt/CPsuite-R81.20/fw1/bin/fw_full
08087000-08089000 rw-p 0003e000 fd:01 17622430                           /opt/CPsuite-R81.20/fw1/bin/fw_full
08089000-080cc000 rw-p 00000000 00:00 0
085fe000-0884c000 rw-p 00000000 00:00 0                                  [heap]
e9e00000-e9e21000 rw-p 00000000 00:00 0
e9e21000-e9f00000 ---p 00000000 00:00 0
ea000000-ea200000 rw-p 00000000 00:00 0
ea3b8000-ea3c6000 r-xp 00000000 fd:01 2002                               /usr/lib/libnss_files-2.17.so
ea3c6000-ea3c7000 r--p 0000d000 fd:01 2002                               /usr/lib/libnss_files-2.17.so
ea3c7000-ea3c8000 rw-p 0000e000 fd:01 2002                               /usr/lib/libnss_files-2.17.so
ea3c8000-ea3ce000 rw-p 00000000 00:00 0
ea3da000-ea3ec000 rw-p 00000000 00:00 0
ea3ec000-ea3ee000 r-xp 00000000 fd:01 653867                             /opt/CPshrd-R81.20/lib/libsicOverSslCrl.so
ea3ee000-ea3f0000 rw-p 00001000 fd:01 653867                             /opt/CPshrd-R81.20/lib/libsicOverSslCrl.so
ea3f0000-ea3f1000 rw-p 00000000 00:00 0
ea3f1000-ea404000 r-xp 00000000 fd:01 653870                             /opt/CPshrd-R81.20/lib/libsic_conf.so
ea404000-ea406000 rw-p 00012000 fd:01 653870                             /opt/CPshrd-R81.20/lib/libsic_conf.so
ea406000-ea43a000 r-xp 00000000 fd:01 50687953                           /opt/CPsuite-R81.20/fw1/lib/libcpc_compiler.so
ea43a000-ea43c000 rw-p 00033000 fd:01 50687953                           /opt/CPsuite-R81.20/fw1/lib/libcpc_compiler.so
ea43c000-ea44d000 rw-p 00000000 00:00 0
ea44d000-ea451000 r-xp 00000000 fd:01 653868                             /opt/CPshrd-R81.20/lib/libsicOverSslExternalDomains.so
ea451000-ea452000 ---p 00004000 fd:01 653868                             /opt/CPshrd-R81.20/lib/libsicOverSslExternalDomains.so
ea452000-ea454000 rw-p 00004000 fd:01 653868                             /opt/CPshrd-R81.20/lib/libsicOverSslExternalDomains.so
ea454000-ea50e000 r-xp 00000000 fd:01 50687777                           /opt/CPsuite-R81.20/fw1/lib/libCPSet2SQL.so
ea50e000-ea511000 r--p 000b9000 fd:01 50687777                           /opt/CPsuite-R81.20/fw1/lib/libCPSet2SQL.so
ea511000-ea514000 rw-p 000bc000 fd:01 50687777                           /opt/CPsuite-R81.20/fw1/lib/libCPSet2SQL.so
ea514000-ea515000 rw-p 00000000 00:00 0
ea515000-ea521000 r-xp 00000000 fd:01 50687824                           /opt/CPsuite-R81.20/fw1/lib/libExtDAO.so
ea521000-ea522000 ---p 0000c000 fd:01 50687824                           /opt/CPsuite-R81.20/fw1/lib/libExtDAO.so
ea522000-ea524000 rw-p 0000c000 fd:01 50687824                           /opt/CPsuite-R81.20/fw1/lib/libExtDAO.so
ea524000-ea533000 r-xp 00000000 fd:01 50687798                           /opt/CPsuite-R81.20/fw1/lib/libCpmixServerAdapter.so
ea533000-ea535000 rw-p 0000e000 fd:01 50687798                           /opt/CPsuite-R81.20/fw1/lib/libCpmixServerAdapter.so
ea535000-ea590000 r-xp 00000000 fd:01 50688002                           /opt/CPsuite-R81.20/fw1/lib/libcpmix.so
ea590000-ea591000 ---p 0005b000 fd:01 50688002                           /opt/CPsuite-R81.20/fw1/lib/libcpmix.so
ea591000-ea593000 rw-p 0005b000 fd:01 50688002                           /opt/CPsuite-R81.20/fw1/lib/libcpmix.so
ea593000-ea594000 rw-p 00000000 00:00 0
ea594000-ea5dd000 r-xp 00000000 fd:01 653856                             /opt/CPshrd-R81.20/lib/libobjlibclient.so
ea5dd000-ea5de000 r--p 00048000 fd:01 653856                             /opt/CPshrd-R81.20/lib/libobjlibclient.so
ea5de000-ea5e0000 rw-p 00049000 fd:01 653856                             /opt/CPshrd-R81.20/lib/libobjlibclient.so
ea5e0000-ea5ee000 rw-p 00000000 00:00 0
ea5ee000-ea5fa000 r-xp 00000000 fd:01 850539                             /opt/CPshrd-R81.20/lib/libcpopenssh.so
ea5fa000-ea5fc000 rw-p 0000b000 fd:01 850539                             /opt/CPshrd-R81.20/lib/libcpopenssh.so
ea5fc000-ea5fe000 r-xp 00000000 fd:01 603062                             /opt/CPshrd-R81.20/lib/libcpssh_statistics.so
ea5fe000-ea600000 rw-p 00001000 fd:01 603062                             /opt/CPshrd-R81.20/lib/libcpssh_statistics.so
ea600000-ea601000 rw-p 00000000 00:00 0
ea601000-ea62a000 r-xp 00000000 fd:01 653771                             /opt/CPshrd-R81.20/lib/libgsoapcpp.so
ea62a000-ea62c000 rw-p 00028000 fd:01 653771                             /opt/CPshrd-R81.20/lib/libgsoapcpp.so
ea62c000-ea635000 r-xp 00000000 fd:01 653866                             /opt/CPshrd-R81.20/lib/libsicOverSsl.so
ea635000-ea637000 rw-p 00008000 fd:01 653866                             /opt/CPshrd-R81.20/lib/libsicOverSsl.so
ea637000-ea64c000 r-xp 00000000 fd:01 50687961                           /opt/CPsuite-R81.20/fw1/lib/libcpc_subr.so
ea64c000-ea64e000 rw-p 00014000 fd:01 50687961                           /opt/CPsuite-R81.20/fw1/lib/libcpc_subr.so
ea64e000-ea690000 rw-p 00000000 00:00 0
ea690000-ea692000 r-xp 00000000 fd:01 50687958                           /opt/CPsuite-R81.20/fw1/lib/libcpc_interface_usr.so
ea692000-ea694000 rw-p 00001000 fd:01 50687958                           /opt/CPsuite-R81.20/fw1/lib/libcpc_interface_usr.so
ea694000-ea6e0000 r-xp 00000000 fd:01 50687952                           /opt/CPsuite-R81.20/fw1/lib/libcpc_builtins.so
ea6e0000-ea6e3000 rw-p 0004b000 fd:01 50687952                           /opt/CPsuite-R81.20/fw1/lib/libcpc_builtins.so
ea6e3000-ea706000 rw-p 00000000 00:00 0
ea706000-ea70e000 r-xp 00000000 fd:01 50687957                           /opt/CPsuite-R81.20/fw1/lib/libcpc_interface_kvm2.so

Any insight would be appreciated.

Best regards,

[G_W_Albrecht]

I would suggest to involve CP TAC ! I only found a similar error here: sk182173: Linux Endpoint computer cannot connect to Security Gateway using SSL Network Extender (SNX...

[RemoteUser]

Hi @G_W_Albrecht , i've already involved TAC,But I don't understand how this SK can be related to this error

[G_W_Albrecht]

Because of the error message in SK "corrupted size vs. prev_size". You have *** Error in `fw': corrupted size vs. prev_size: 0x0879db68 *** But i do not think this SK will help as you do not have a SNX issue...

[RemoteUser]

I think i found the solution:

PRJ-60852

Security Gateway

In some scenarios, when working with SecureXL in User Mode (UPPAK) and enabling a debug filter, the Security Gateway may crash.

[the_rock]

That definitely sounds the same.

[G_W_Albrecht]

Did you upgrade to JT 101 already and did that resolve the issue ?

[Lesley]

Other commands still work? I think issue is not only with fw ctl zedbug but bigger. Maybe check stuff like cphaprob stat , hcp -r all

[RemoteUser]

hey buddy,

I found this running hcp:
Result: ERROR |
| |
| Description: This test checks if there are any user mode core dumps and if possible, prints their backtrace. |
| |
| Summary:Found 1 coredump on the machine |
| |
| +-----------------------------------------------------+ |
| | User Mode coredumps | |
| +--------------+------+---------+---------------------+ |
| | Process Name | PID | Size | Creation Date | |
| +==============+======+=========+=====================+ |
| | fw_full | 3265 | 6.46 MB | 2025-05-22 15:23:49 | |
| +--------------+------+---------+---------------------+ |
| |
| Finding: |
| Process fw_full crashed on 2025-05-22 15:23:49 |
| |
| Suggested solutions: |
| - Please contact Check Point Support and provide the file /var/log/dump/usermode/fw_full.3265.core.gz |
| - If you want to see core dumps backtraces, please ask Check Point Support |
| to install GDB under bin folder |
|

[Bob_Zimmerman]

This is the immediate problem. The process crashed when you ended the debug. You need to open a ticket and send the core dump and a cpinfo to the TAC. They should be able to tell pretty quickly if the core dump is a known issue or not.

[RemoteUser]

I have already sent everything, I am waiting for a response....

[Chris_Atkinson]

Can you please share additional context i.e. appliance model and JHF take?

[RemoteUser]

Platform: RS-20-00
Model: Check Point 9200
CPU Model: 13th Gen Intel(R) Core(TM) i3-13100E
CPU Frequency: 3300.000 Mhz
Number of Cores: 8
CPU Hyperthreading: Enabled

R81.20
JHF > 99 + hotfix

[Timothy_Hall]

What happens if you try just fw ctl zdebug drop without the "+".  The + enables logging of drops in SecureXL/UPPAK.

[RemoteUser]

It seems to be working normally. This firewall uses UPPAK mode, so maybe using the "+" option during debugging could be causing some issues?

[RemoteUser]

The + in the command fw ctl zdebug drop  does not enable drop logging in SecureXL when operating in  (UPPAK).

According toHow to use the fw ctl zdebug command to view drops on the Security Gateway, the + drop option in fw ctl zdebug enables logging of drop messages from the kernel (INSPECT) and also from PPAK (Post-Processing Acceleration Kernel) not from UPPAK.

Or am i wrong???

[the_rock]

Hey bro,

Lets take a step back, as they say...what is the actual issue?

Andy

[RemoteUser]

Hey Buddy
| Finding: |
| Process fw_full crashed on 2025-05-22 15:23:49 |
The real issue is that, as you can see from the output, it looks like the FW module crashed when we tried to run  fw ctl zdebug drop

[the_rock]

Weird indeed...does reboot help?

Andy

[RemoteUser]

Hey Buddy
Unfortunately, I don't think a reboot will help, because there's a module, fw_full, that has crashed, although I haven't tried rebooting yet.

[the_rock]

Personally, if I were you, would do it.

Andy