topic Re: What are fw monitor chain position c and r in Firewall and Security Management

What are fw monitor chain position c and r

Don_Paterson — Thu, 19 Oct 2023 07:31:16 GMT

I have seen chain position c (ic) and r (Ir) and also ia and Iq.

Can anyone explain please?

Attached is from R81.20 (Build 021) where I see ic and Ir.

I have not done exhaustive research and it is not obvious.

Nothing in here:

https://community.checkpoint.com/t5/Security-Gateways/R80-x-Performance-Tuning-and-Debug-Tips-fw-monitor/td-p/41563

or here:

https://support.checkpoint.com/results/sk/sk30583

Re: What are fw monitor chain position c and r

Don_Paterson — Thu, 19 Oct 2023 16:59:47 GMT

Adding to my question (referencing the same attached PDF):

The chain position 3 does not match the actual pre-inbound (i) chain position, which is 12.

It makes sense if I exclude all SecureXL from the fw ctl chain command output and count 'tcpt inbound' as chain position 0, which makes position 12 change to 3.

Is this expected behaviour and if so will that change?

And what about those letters;
Is c the hexadecimal for 12 in this case, meaning that real position 0 (SecureXL stateless check) is actually 0 in a hex count, so that 10 (IP Options Strip) is A and therefore c=12 (pre-inbound)?

Thanks,

Don

Re: What are fw monitor chain position c and r

Don_Paterson — Thu, 19 Oct 2023 17:12:02 GMT

@PhoneBoy @_Val_
Hello chaps, 🙂
Looking for some authoritative clarification please.

Cheers,

Don

Re: What are fw monitor chain position c and r

Bob_Zimmerman — Thu, 19 Oct 2023 18:51:03 GMT

It's not actually hex, but it works similarly. There's only one character for that field. It goes 0 through 9, a through z. I'm not sure what happens if you have more than 34 kernel extensions (z would be after extension 34 or before 35). It might go to A through Z. Lowercase r would be before extension 27.

Re: What are fw monitor chain position c and r

the_rock — Thu, 19 Oct 2023 19:10:12 GMT

If anyone on this planet can give you right answer to this, its probably @Timothy_Hall . I would look in his book which I have, but too busy now lol

Andy

Re: What are fw monitor chain position c and r

Don_Paterson — Thu, 19 Oct 2023 19:39:27 GMT

Thanks.

Interesting that you call them extensions.

I think of them as kernel chain modules.

In the SK the -a switch is described and mentions the positions, implying that the numbers are relevant but not in special CP hex format.

That might need a review for clarification.

Is it documented anywhere that they are labeled by single character and the system/method?

https://support.checkpoint.com/results/sk/sk30583

If the captured data is saved into an output file (using the "-o <output_file_name>" switch), one of the fields written into the output file would be the chain position of the FW Monitor chain module.
Together with a simultaneous execution of "fw ctl chain" command you can determine where the packet was captured. Especially when using "-p all" switch, you will find the same packet captured multiples times at different chain positions.

-ci count
-co count

Captures a specific number of packets.

Re: What are fw monitor chain position c and r

Bob_Zimmerman — Thu, 19 Oct 2023 20:47:14 GMT

Let's just say I'm intimately familiar with network kernels.

The first character in fw monitor's position data is coarse with just the four possible values. The second character gives you the exact number in the chain for finer positional information. I'm not really sure why this is represented as an ASCII character rather than as a raw byte. As features are enabled and disabled, extensions are attached to and detached from the network kernel, so the number of a given module can change.

I had to look it up, and the proper term for base 36 is actually "hexatrigesimal".

Re: What are fw monitor chain position c and r

PhoneBoy — Thu, 19 Oct 2023 21:34:40 GMT

It's also not in the Product Documentation either.
I know 'q' is QoS.
Not sure about the others, but I will see if we can get that information and update the docs.

Re: What are fw monitor chain position c and r

G_W_Albrecht — Fri, 20 Oct 2023 06:45:00 GMT

Open an informative SR# with CP TAC !

Re: What are fw monitor chain position c and r

Don_Paterson — Fri, 20 Oct 2023 07:09:06 GMT

🙂

I prefer to keep it here in Check Mates. Nice and lively. 😉

Re: What are fw monitor chain position c and r

Don_Paterson — Fri, 20 Oct 2023 07:13:40 GMT

In this case, and probably most others it is likely to be the hexatrigesimal system in the Kernel design.
I am not running QoS in this scenario.
Thanks @Bob_Zimmerman

https://trustconverter.com/en/base-number-conversion/hexatrigesimal/hexatrigesimal-to-decimal.html

Re: What are fw monitor chain position c and r

Timothy_Hall — Fri, 20 Oct 2023 19:21:56 GMT

All the capture examples you gave are of traffic terminating at the firewall itself on port 4434 (not transiting to someplace like the Internet) . I know this because the IP addresses you are using are standard for an Authorized Training Center training lab. Do you see these extra letters for traffic transiting the firewall to someplace like the Internet? My guess is you won't.

Theory 1: Some kind of NIC offloading or other kind of hardware acceleration is being indicated by these extra letters. What is the driver type on the interface you took the captures on (ethtool -i)? This could be possible if you are using the Mellanox/Lightspeed cards. I don't see these extra letters in my fw monitor -e captures for the same kind of traffic in my training lab on R81.20 T26 (or R81.20 GA) with the vmxnet3 driver.

Theory 2: All traffic terminating at the firewall itself must be handled in the slowpath and is ineligible for any kind of acceleration. In the old days when the fwmonitor chain module was inserted in the list of modules it was right at the top. However in R80.20+, SecureXL has it's own "chain modules" now and as a result you can see the "i" fwmonitor has been pushed way down to number 12, so all kinds of things can happen prior to the "i" capture point. Normally slowpath traffic must go through all chain modules. But for the special case of traffic terminating at the gateway itself I'm wondering if the extra letter is indicating where a "skip" of chain modules ended, or more precisely from what chain module the fwmonitor module received it from (or sent it to) using the hexatrigesimal numbering system Bob mentioned. There is no point in going through the SecureXL-based chain modules (and a few others) for non-transiting traffic to and from the firewall itself. So either this "skip" is some new optimization, or has been always happening in past versions but fw monitor has now been updated to show us what is happening, because fwmonitor is so far from the "top" of the chain module sequence now and so much can happen before that "i" point (and others).

Or that second theory could be totally wrong, it is just a guess.

Re: What are fw monitor chain position c and r

Bob_Zimmerman — Fri, 20 Oct 2023 20:10:41 GMT

It's pretty easy to see the base-36 counting in a -p all capture taken on a lab firewall. I just did it on one of mine and I got i0 (sxl_state_check), i3 (sxl_lookup), i9 (ipopt_strip), ia (asm), ib (fw multik misc proto forwarding), ic (fw), Id (scv), Ie (offload_in), If (post_vm), Ig (pass_str), Ih (cpas), Ii (ipopt_res), Ij on the inbound leg. j (19) is after the last inbound extension. All the other capture points are at the input to the extension.

Then on the outbound, I have o0 (ipopt_strip), o1 (cpas), o2 (pass_str), o3 (asm), o4 (fw), O5 (post_vm), O6 (cpas), O7 (ipopt_res), Od. My outbound chain has 12 extensions, so d (13) is the capture point after the last one.

[Expert@DallasSA]# fw ctl chain in chain (19): 0: -7fffffff (0000000000000000) (00000000) SecureXL stateless check (sxl_state_check) 1: -7ffffffe (0000000000000000) (00000000) SecureXL VPN before decryption (vpn_in_before_decrypt) 2: -7ffffffd (0000000000000000) (00000000) SecureXL VPN after decryption (vpn_in_after_decrypt) 3: 6 (0000000000000000) (00000000) SecureXL lookup (sxl_lookup) 4: 7 (0000000000000000) (00000000) SecureXL QOS inbound (sxl_qos_inbound) 5: 8 (0000000000000000) (00000000) SecureXL inbound (sxl_inbound) 6: 9 (0000000000000000) (00000000) SecureXL medium path streaming (sxl_medium_path_streaming) 7: 10 (0000000000000000) (00000000) SecureXL inline path streaming (sxl_inline_path_streaming) 8: 11 (0000000000000000) (00000000) SecureXL Routing (sxl_routing) 9: -7f800000 (ffffffff931903f0) (ffffffff) IP Options Strip (in) (ipopt_strip) 10: - 1fffff8 (ffffffff9318dc40) (00000001) Stateless verifications (in) (asm) 11: - 1fffff7 (ffffffff93121720) (00000001) fw multik misc proto forwarding 12: 0 (ffffffff93d58910) (00000001) fw VM inbound (fw) 13: 2 (ffffffff93195900) (00000001) fw SCV inbound (scv) 14: 5 (ffffffff92eb4880) (00000003) fw offload inbound (offload_in) 15: 20 (ffffffff93d5c0f0) (00000001) fw post VM inbound (post_vm) 16: 7f730000 (ffffffff93224740) (00000001) passive streaming (in) (pass_str) 17: 7f750000 (ffffffff93aaa0c0) (00000001) TCP streaming (in) (cpas) 18: 7f800000 (ffffffff93190380) (ffffffff) IP Options Restore (in) (ipopt_res) out chain (13): 0: -7f800000 (ffffffff931903f0) (ffffffff) IP Options Strip (out) (ipopt_strip) 1: - 1fffff0 (ffffffff93aa74d0) (00000001) TCP streaming (out) (cpas) 2: - 1ffff50 (ffffffff93224740) (00000001) passive streaming (out) (pass_str) 3: - 1f00000 (ffffffff9318dc40) (00000001) Stateless verifications (out) (asm) 4: 0 (ffffffff93d58910) (00000001) fw VM outbound (fw) 5: 10 (ffffffff93d5c0f0) (00000001) fw post VM outbound (post_vm) 6: 7f700000 (ffffffff93aa79c0) (00000001) TCP streaming post VM (cpas) 7: 7f800000 (ffffffff93190380) (ffffffff) IP Options Restore (out) (ipopt_res) 8: 7f900000 (0000000000000000) (00000000) SecureXL outbound (sxl_outbound) 9: 7fa00000 (0000000000000000) (00000000) SecureXL QOS outbound (sxl_qos_outbound) 10: 7fb00000 (0000000000000000) (00000000) SecureXL VPN before encryption (vpn_in_before_encrypt) 11: 7fc00000 (0000000000000000) (00000000) SecureXL VPN after encryption (vpn_in_after_encrypt) 12: 7fd00000 (0000000000000000) (00000000) SecureXL Deliver (sxl_deliver)

Admittedly, my firewall is R81.10, but R81.20 is not fundamentally different in this regard. I also haven't ever actually seen letters higher than r, so I'm speculating it goes to z. Seems like reasonable speculation to me, though.

The "fwmonitor (i/f side)" being down at 12 in Don's chain is related to why fw monitors now go i-i-I-o-O with an extra little-i. When you run one to the terminal instead of to a file, you can see the first little-i is in ppak_#. I think the number is a dispatcher thread ID, but I haven't looked into it in depth. The second little-i and all the other points are in fw_#, and the # is a CoreXL worker thread ID.

An fw monitor -p all in my terminal showing a SYN to the firewall and SYN-ACK response from the firewall:

[vs_0][ppak_0] eth1:i0 (SecureXL stateless check)[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][ppak_0] eth1:i3 (SecureXL lookup)[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:i9 (IP Options Strip (in))[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:i10 (Stateless verifications (in))[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:i11 (fw multik misc proto forwarding)[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:i12 (fw VM inbound )[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:I13 (fw SCV inbound)[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:I14 (fw offload inbound)[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:I15 (fw post VM inbound )[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:I16 (passive streaming (in))[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:I17 (TCP streaming (in))[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:I18 (IP Options Restore (in))[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:I19 (Chain End)[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 62915 -> 443 .S.... seq=fb68e894 ack=00000000 [vs_0][fw_1] eth1:o0 (IP Options Strip (out))[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:o1 (TCP streaming (out))[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:o2 (passive streaming (out))[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:o3 (Stateless verifications (out))[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:o4 (fw VM outbound)[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:O5 (fw post VM outbound )[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:O6 (TCP streaming post VM)[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:O7 (IP Options Restore (out))[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895 [vs_0][fw_1] eth1:O13 (Chain End)[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 62915 .S..A. seq=54fb4a48 ack=fb68e895

And an fw monitor without -p all showing the a similar SYN and SYN-ACK:

[vs_0][ppak_0] eth1:i[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 63155 -> 443 .S.... seq=7ef1758f ack=00000000 [vs_0][fw_1] eth1:i[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 63155 -> 443 .S.... seq=7ef1758f ack=00000000 [vs_0][fw_1] eth1:I[64]: 10.0.3.26 -> 10.0.1.253 (TCP) len=64 id=0 TCP: 63155 -> 443 .S.... seq=7ef1758f ack=00000000 [vs_0][fw_1] eth1:o[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 63155 .S..A. seq=b9e77874 ack=7ef17590 [vs_0][fw_1] eth1:O[60]: 10.0.1.253 -> 10.0.3.26 (TCP) len=60 id=0 TCP: 443 -> 63155 .S..A. seq=b9e77874 ack=7ef17590

Note that the capture without the -p all with output to the terminal doesn't show the detailed position indicator. If you capture to a file, it does record the detailed position. In my case, I get i3, ia, Ih, o1, O7.

Re: What are fw monitor chain position c and r

PhoneBoy — Fri, 20 Oct 2023 22:18:34 GMT

Yes, I confirmed with R&D that this is exactly what's happening.
When you use the -a switch, instead of the more familiar letters, you'll get chain position in hexatrigesimal format (base 36).
We'll get the SK updated.

Re: What are fw monitor chain position c and r

the_rock — Fri, 20 Oct 2023 22:58:19 GMT

@PhoneBoy to the rescue, as always! By the way, Im still laughing about this...so today, had a call with TAC for some weird routing issue and I bet guy was brand new to TAC (though he was pretty good), but he called you Paperboy when I mentioned your nickname about some other community post.

Paperboy, haha...though you have always been know as Phoneboy, so we will stick with that : - )

Andy

Re: What are fw monitor chain position c and r

PhoneBoy — Sat, 21 Oct 2023 00:49:41 GMT

I've been called many things by many people. 😉

This is what I think of when I hear the name Paperboy:

Re: What are fw monitor chain position c and r

the_rock — Sat, 21 Oct 2023 00:55:16 GMT

Good one lol.

Re: What are fw monitor chain position c and r

genisis__ — Sat, 21 Oct 2023 22:05:58 GMT

I had that game!

Re: What are fw monitor chain position c and r

HeikoAnkenbrand — Sun, 22 Oct 2023 13:23:40 GMT

Hi guys (@Don_Paterson, @Bob_Zimmerman)

A tip from me!

At the moment, the Wireshark plugin actually has a bug and displays the chain modules after 9 as ASCII.

--------

"fw monitor" is actually a debug command that can be used to analyse the packet flow in SecureXL and the Firewall kernel. What Check Point doesn't tell you in the manuals is, that this is actually a kernel debug.
You could also analyse this with the following commands and get much more info here:

For example with "fw ctl zdebug". Filters can be set as with fw monitor:
# fw ctl zdebug + fw conn -e "accept(host(8.8.8.8));"

Or if the debug buffer of 1MB is not enough, you can do a debug:

Session 1# fw ctl debug -m fw conn -e "accept(host(8.8.8.8));"
Session 1# fw ctl kdebug -f -T > /var/log/test.txt

Session 2# tail -f /var/log/test.txt

Here you can see all the real inspection points (ppak,i,I,o,O,...) and even more information about the connections.

For example, if you would like to have information about the NAT, you can do the following:

# fw ctl zdebug + fw conn xlate xltrc nat -e "accept(host(8.8.8.8));"

Because "fw monitor", "fw ctl debug" and "fw ctl zdebug" are kernel debugs, you should always be a little careful from a performance point of view and don't forget to disable the debug if necessary;-). "fw ctl zdebug" and "fw monitor" automatically disable the debug.

Re: What are fw monitor chain position c and r

Bob_Zimmerman — Sun, 22 Oct 2023 14:28:43 GMT

If it is a bug, it's not one in Wireshark. Take a look at an fw monitor file with xxd and you can clearly see the position encoded as two bytes with ASCII character values:

[Expert@DallasSA]# xxd pAll.snoop 0000000: 736e 6f6f 7000 0000 0000 0002 0000 0004 snoop........... 0000010: 0000 004e 0000 004e 0000 0068 0000 0000 ...N...N...h.... 0000020: 6535 2794 0008 c5ef 6930 6574 6831 0000 e5'.....i0eth1.. 0000030: 3030 6630 0800 4500 0040 0000 4000 3f06 00f0..E..@..@.?. 0000040: 22a2 0a00 031a 0a00 01fd c229 01bb 34a5 "..........)..4. 0000050: 1a05 0000 0000 b002 ffff 1328 0000 0204 ...........(.... 0000060: 2300 0103 0306 0101 080a 295f b183 0000 #.........)_.... 0000070: 0000 0402 0000 0000 0000 004e 0000 004e ...........N...N 0000080: 0000 0068 0000 0000 6535 2794 0008 c5fa ...h....e5'..... 0000090: 6933 6574 6831 0000 3030 6630 0800 4500 i3eth1..00f0..E. 00000a0: 0040 0000 4000 3f06 22a2 0a00 031a 0a00 .@..@.?."....... 00000b0: 01fd c229 01bb 34a5 1a05 0000 0000 b002 ...)..4......... 00000c0: ffff 1328 0000 0204 2300 0103 0306 0101 ...(....#....... 00000d0: 080a 295f b183 0000 0000 0402 0000 0000 ..)_............ 00000e0: 0000 004e 0000 004e 0000 0068 0000 0000 ...N...N...h.... 00000f0: 6535 2794 0008 c6a4 6939 6574 6831 0000 e5'.....i9eth1.. 0000100: 3030 6630 0800 4500 0040 0000 4000 3f06 00f0..E..@..@.?. 0000110: 22a2 0a00 031a 0a00 01fd c229 01bb 34a5 "..........)..4. 0000120: 1a05 0000 0000 b002 ffff 1328 0000 0204 ...........(.... 0000130: 2300 0103 0306 0101 080a 295f b183 0000 #.........)_.... 0000140: 0000 0402 0000 0000 0000 004e 0000 004e ...........N...N 0000150: 0000 0068 0000 0000 6535 2794 0008 c6a9 ...h....e5'..... 0000160: 6961 6574 6831 0000 3030 6630 0800 4500 iaeth1..00f0..E.

Offset 0x28:2: 0x6930

Offset 0x90:2: 0x6933

Offset 0xf8:2: 0x6939

Offset 0x160:2: 0x6961

In ASCII, 0x30 to 0x39 are digits 0 through 9, 0x41 through 0x5a are uppercase A through Z, and 0x61 through 0x7a are lowercase a through z. You can see this without consulting an ASCII table in the ASCII decode of the raw bytes on the right. If the second byte wasn't meant to be interpreted as ASCII, why use ASCII values, including the big discontinuity from 0x39 to 0x61?

IP addresses are represented as raw values, as you can see as offset 0x32:8. 0x0a00031a decoded as a dotted decimal IP address corresponds to 10.0.3.26, which is my client. 0x0a0001fd corresponds to 10.0.1.253, which is this standalone's address.