topic Unusual zdebug Output in General Topics

Unusual zdebug Output

RemoteUser — Fri, 23 May 2025 07:26:08 GMT

Hi everyone,
I hope you're all doing well.

The client recently reported a rather unusual error that I have to admit I've never seen before. After running a zdebug drop on the firewall to monitor dropped packets, the following output was observed:

Has anyone come across something similar?

I'm wondering if this behavior could be related to the fact that SecureXL is running in UPPAK mode. Could this debugging method possibly interfere or produce unexpected results in such a configuration?

fw ctl zdebug + drop | grep x.x.x.x
^C
Next time perform for exit: "fw ctl debug 0"

cpdev_user_ioctl: ioctl failed to device /vs0/cp/usim/main
: Interrupted system call
*** Error in `fw': corrupted size vs. prev_size: 0x0879db68 ***
======= Backtrace: =========
/lib/libc.so.6(+0x78eaf)[0xefbaaeaf]
/lib/libc.so.6(__libc_calloc+0xb3)[0xefbad113]
/opt/CPshrd-R81.20/lib/libOS.so(+0x464c6)[0xf06344c6]
/opt/CPshrd-R81.20/lib/libOS.so(+0x46a07)[0xf0634a07]
/opt/CPshrd-R81.20/lib/libOS.so(CpSystemEx+0x161)[0xf0634f31]
/opt/CPshrd-R81.20/lib/libOS.so(CpSystemWithFlags+0x2b)[0xf063513b]
/opt/CPshrd-R81.20/lib/libkiss_apps.so(kiss_debug_command+0x14f7)[0xf1649eb7]
/opt/CPshrd-R81.20/lib/libkiss_apps.so(kiss_zdebug_command+0x426)[0xf164e046]
/opt/CPsuite-R81.20/fw1/lib/libfw1.so(+0x118f3f)[0xf38b3f3f]
/opt/CPsuite-R81.20/fw1/lib/libfw1.so(main_fwctl+0x4b4)[0xf38b7674]
/opt/CPsuite-R81.20/fw1/lib/libfw1.so(fw_cmain+0xccf)[0xf38fca2f]
fw(main+0xa8)[0x8052b48]
/lib/libc.so.6(__libc_start_main+0xde)[0xefb4a3ae]
fw[0x8052be5]
======= Memory map: ========
08048000-08086000 r-xp 00000000 fd:01 17622430                           /opt/CPsuite-R81.20/fw1/bin/fw_full
08086000-08087000 r--p 0003d000 fd:01 17622430                           /opt/CPsuite-R81.20/fw1/bin/fw_full
08087000-08089000 rw-p 0003e000 fd:01 17622430                           /opt/CPsuite-R81.20/fw1/bin/fw_full
08089000-080cc000 rw-p 00000000 00:00 0
085fe000-0884c000 rw-p 00000000 00:00 0                                  [heap]
e9e00000-e9e21000 rw-p 00000000 00:00 0
e9e21000-e9f00000 ---p 00000000 00:00 0
ea000000-ea200000 rw-p 00000000 00:00 0
ea3b8000-ea3c6000 r-xp 00000000 fd:01 2002                               /usr/lib/libnss_files-2.17.so
ea3c6000-ea3c7000 r--p 0000d000 fd:01 2002                               /usr/lib/libnss_files-2.17.so
ea3c7000-ea3c8000 rw-p 0000e000 fd:01 2002                               /usr/lib/libnss_files-2.17.so
ea3c8000-ea3ce000 rw-p 00000000 00:00 0
ea3da000-ea3ec000 rw-p 00000000 00:00 0
ea3ec000-ea3ee000 r-xp 00000000 fd:01 653867                             /opt/CPshrd-R81.20/lib/libsicOverSslCrl.so
ea3ee000-ea3f0000 rw-p 00001000 fd:01 653867                             /opt/CPshrd-R81.20/lib/libsicOverSslCrl.so
ea3f0000-ea3f1000 rw-p 00000000 00:00 0
ea3f1000-ea404000 r-xp 00000000 fd:01 653870                             /opt/CPshrd-R81.20/lib/libsic_conf.so
ea404000-ea406000 rw-p 00012000 fd:01 653870                             /opt/CPshrd-R81.20/lib/libsic_conf.so
ea406000-ea43a000 r-xp 00000000 fd:01 50687953                           /opt/CPsuite-R81.20/fw1/lib/libcpc_compiler.so
ea43a000-ea43c000 rw-p 00033000 fd:01 50687953                           /opt/CPsuite-R81.20/fw1/lib/libcpc_compiler.so
ea43c000-ea44d000 rw-p 00000000 00:00 0
ea44d000-ea451000 r-xp 00000000 fd:01 653868                             /opt/CPshrd-R81.20/lib/libsicOverSslExternalDomains.so
ea451000-ea452000 ---p 00004000 fd:01 653868                             /opt/CPshrd-R81.20/lib/libsicOverSslExternalDomains.so
ea452000-ea454000 rw-p 00004000 fd:01 653868                             /opt/CPshrd-R81.20/lib/libsicOverSslExternalDomains.so
ea454000-ea50e000 r-xp 00000000 fd:01 50687777                           /opt/CPsuite-R81.20/fw1/lib/libCPSet2SQL.so
ea50e000-ea511000 r--p 000b9000 fd:01 50687777                           /opt/CPsuite-R81.20/fw1/lib/libCPSet2SQL.so
ea511000-ea514000 rw-p 000bc000 fd:01 50687777                           /opt/CPsuite-R81.20/fw1/lib/libCPSet2SQL.so
ea514000-ea515000 rw-p 00000000 00:00 0
ea515000-ea521000 r-xp 00000000 fd:01 50687824                           /opt/CPsuite-R81.20/fw1/lib/libExtDAO.so
ea521000-ea522000 ---p 0000c000 fd:01 50687824                           /opt/CPsuite-R81.20/fw1/lib/libExtDAO.so
ea522000-ea524000 rw-p 0000c000 fd:01 50687824                           /opt/CPsuite-R81.20/fw1/lib/libExtDAO.so
ea524000-ea533000 r-xp 00000000 fd:01 50687798                           /opt/CPsuite-R81.20/fw1/lib/libCpmixServerAdapter.so
ea533000-ea535000 rw-p 0000e000 fd:01 50687798                           /opt/CPsuite-R81.20/fw1/lib/libCpmixServerAdapter.so
ea535000-ea590000 r-xp 00000000 fd:01 50688002                           /opt/CPsuite-R81.20/fw1/lib/libcpmix.so
ea590000-ea591000 ---p 0005b000 fd:01 50688002                           /opt/CPsuite-R81.20/fw1/lib/libcpmix.so
ea591000-ea593000 rw-p 0005b000 fd:01 50688002                           /opt/CPsuite-R81.20/fw1/lib/libcpmix.so
ea593000-ea594000 rw-p 00000000 00:00 0
ea594000-ea5dd000 r-xp 00000000 fd:01 653856                             /opt/CPshrd-R81.20/lib/libobjlibclient.so
ea5dd000-ea5de000 r--p 00048000 fd:01 653856                             /opt/CPshrd-R81.20/lib/libobjlibclient.so
ea5de000-ea5e0000 rw-p 00049000 fd:01 653856                             /opt/CPshrd-R81.20/lib/libobjlibclient.so
ea5e0000-ea5ee000 rw-p 00000000 00:00 0
ea5ee000-ea5fa000 r-xp 00000000 fd:01 850539                             /opt/CPshrd-R81.20/lib/libcpopenssh.so
ea5fa000-ea5fc000 rw-p 0000b000 fd:01 850539                             /opt/CPshrd-R81.20/lib/libcpopenssh.so
ea5fc000-ea5fe000 r-xp 00000000 fd:01 603062                             /opt/CPshrd-R81.20/lib/libcpssh_statistics.so
ea5fe000-ea600000 rw-p 00001000 fd:01 603062                             /opt/CPshrd-R81.20/lib/libcpssh_statistics.so
ea600000-ea601000 rw-p 00000000 00:00 0
ea601000-ea62a000 r-xp 00000000 fd:01 653771                             /opt/CPshrd-R81.20/lib/libgsoapcpp.so
ea62a000-ea62c000 rw-p 00028000 fd:01 653771                             /opt/CPshrd-R81.20/lib/libgsoapcpp.so
ea62c000-ea635000 r-xp 00000000 fd:01 653866                             /opt/CPshrd-R81.20/lib/libsicOverSsl.so
ea635000-ea637000 rw-p 00008000 fd:01 653866                             /opt/CPshrd-R81.20/lib/libsicOverSsl.so
ea637000-ea64c000 r-xp 00000000 fd:01 50687961                           /opt/CPsuite-R81.20/fw1/lib/libcpc_subr.so
ea64c000-ea64e000 rw-p 00014000 fd:01 50687961                           /opt/CPsuite-R81.20/fw1/lib/libcpc_subr.so
ea64e000-ea690000 rw-p 00000000 00:00 0
ea690000-ea692000 r-xp 00000000 fd:01 50687958                           /opt/CPsuite-R81.20/fw1/lib/libcpc_interface_usr.so
ea692000-ea694000 rw-p 00001000 fd:01 50687958                           /opt/CPsuite-R81.20/fw1/lib/libcpc_interface_usr.so
ea694000-ea6e0000 r-xp 00000000 fd:01 50687952                           /opt/CPsuite-R81.20/fw1/lib/libcpc_builtins.so
ea6e0000-ea6e3000 rw-p 0004b000 fd:01 50687952                           /opt/CPsuite-R81.20/fw1/lib/libcpc_builtins.so
ea6e3000-ea706000 rw-p 00000000 00:00 0
ea706000-ea70e000 r-xp 00000000 fd:01 50687957                           /opt/CPsuite-R81.20/fw1/lib/libcpc_interface_kvm2.so

Any insight would be appreciated.

Best regards,

Re: Unusual zdebug Output

G_W_Albrecht — Fri, 23 May 2025 07:45:19 GMT

I would suggest to involve CP TAC ! I only found a similar error here: sk182173: Linux Endpoint computer cannot connect to Security Gateway using SSL Network Extender (SNX) version 80008407

Re: Unusual zdebug Output

RemoteUser — Fri, 23 May 2025 07:48:34 GMT

Hi @G_W_Albrecht , i've already involved TAC,But I don't understand how this SK can be related to this error

Re: Unusual zdebug Output

G_W_Albrecht — Fri, 23 May 2025 07:57:35 GMT

Because of the error message in SK "corrupted size vs. prev_size". You have *** Error in `fw': corrupted size vs. prev_size: 0x0879db68 *** But i do not think this SK will help as you do not have a SNX issue...

Re: Unusual zdebug Output

Lesley — Fri, 23 May 2025 08:55:33 GMT

Other commands still work? I think issue is not only with fw ctl zedbug but bigger. Maybe check stuff like cphaprob stat , hcp -r all

Re: Unusual zdebug Output

RemoteUser — Fri, 23 May 2025 09:55:30 GMT

Re: Unusual zdebug Output

Chris_Atkinson — Fri, 23 May 2025 10:20:55 GMT

Can you please share additional context i.e. appliance model and JHF take?

Re: Unusual zdebug Output

RemoteUser — Fri, 23 May 2025 10:22:43 GMT

Platform: RS-20-00
Model: Check Point 9200
CPU Model: 13th Gen Intel(R) Core(TM) i3-13100E
CPU Frequency: 3300.000 Mhz
Number of Cores: 8
CPU Hyperthreading: Enabled

R81.20
JHF > 99 + hotfix

Re: Unusual zdebug Output

Timothy_Hall — Fri, 23 May 2025 12:49:27 GMT

What happens if you try just fw ctl zdebug drop without the "+". The + enables logging of drops in SecureXL/UPPAK.

Re: Unusual zdebug Output

Bob_Zimmerman — Fri, 23 May 2025 20:06:16 GMT

This is the immediate problem. The process crashed when you ended the debug. You need to open a ticket and send the core dump and a cpinfo to the TAC. They should be able to tell pretty quickly if the core dump is a known issue or not.

Re: Unusual zdebug Output

the_rock — Sat, 24 May 2025 22:22:04 GMT

Hey bro,

Lets take a step back, as they say...what is the actual issue?

Andy

Re: Unusual zdebug Output

RemoteUser — Sun, 25 May 2025 07:53:51 GMT

Hey Buddy
| Finding: |
| Process fw_full crashed on 2025-05-22 15:23:49 |
The real issue is that, as you can see from the output, it looks like the FW module crashed when we tried to run fw ctl zdebug drop

Re: Unusual zdebug Output

RemoteUser — Sun, 25 May 2025 07:54:30 GMT

I have already sent everything, I am waiting for a response....

Re: Unusual zdebug Output

RemoteUser — Sun, 25 May 2025 08:03:30 GMT

It seems to be working normally. This firewall uses UPPAK mode, so maybe using the "+" option during debugging could be causing some issues?

Re: Unusual zdebug Output

the_rock — Sun, 25 May 2025 13:17:51 GMT

Weird indeed...does reboot help?

Andy

Re: Unusual zdebug Output

RemoteUser — Mon, 26 May 2025 07:51:48 GMT

Hey Buddy
Unfortunately, I don't think a reboot will help, because there's a module, fw_full, that has crashed, although I haven't tried rebooting yet.

Re: Unusual zdebug Output

RemoteUser — Mon, 26 May 2025 07:57:19 GMT

I think i found the solution:

PRJ-60852

Security Gateway

In some scenarios, when working with SecureXL in User Mode (UPPAK) and enabling a debug filter, the Security Gateway may crash.

Re: Unusual zdebug Output

RemoteUser — Mon, 26 May 2025 08:08:03 GMT

The + in the command fw ctl zdebug drop does not enable drop logging in SecureXL when operating in (UPPAK).

According toHow to use the fw ctl zdebug command to view drops on the Security Gateway, the + drop option in fw ctl zdebug enables logging of drop messages from the kernel (INSPECT) and also from PPAK (Post-Processing Acceleration Kernel) not from UPPAK.

Or am i wrong???

Re: Unusual zdebug Output

the_rock — Mon, 26 May 2025 11:45:42 GMT

Personally, if I were you, would do it.

Andy

Re: Unusual zdebug Output

the_rock — Mon, 26 May 2025 11:46:43 GMT

That definitely sounds the same.