SOLVED: Disappearing packets / Incoming S2S VPN (Harmony SASE)

[Masek]

I have a centrally managed Check Point Spark 1550. Central Management is R81.20 Take 65, the 1550 uses R81.10.10.

For using Harmony SASE I setup a S2S VPN following this guide: https://support.perimeter81.com/docs/configuring-perimeter-site-to-site-with-check-point-firewall-us....

This generally works, but sometimes incoming packets just disappear.

In "fw monitor" it looks like this:

[vs_0][fw_1] WAN2:i9 (IP Options Strip (in))[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i10 (vpn multik forward in)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i11 (vpn decrypt)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i12 (l2tp inbound)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i13 (Stateless verifications (in))[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i14 (fw multik misc proto forwarding)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i15 (vpn tagging inbound)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i16 (vpn decrypt verify)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:i17 (fw VM inbound )[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:I18 (vpn policy inbound)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:I19 (vpn before offload)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

[vs_0][fw_1] WAN2:I20 (fw offload inbound)[44]: 10.2.3.254 -> 10.0.1.124 (TCP) len=60 id=40353TCP: 40914 -> 80 .S.... seq=d8f1a09d ack=00000000

After that, no more is seen from this packet. I checked if it was NATed, but it is just gone. I also checked if it was going through anyway, but it didn't make it to the outgoing interface.

According to the log the connection has been accepted.

This only affects packets that come from a service within Harmony SASE (DNS, Published aplication) which all use the 10.2.3.254 address. Packets from clients connected via Harmony SASE are not affected.

I have to confess, I am a bit baffled.

[Masek]

I think I found it. Now I only need to find out what it means:

@;1756498;[cpu_2];[fw4_2];fw_log_drop_ex: Packet proto=6 10.2.3.254:47320 -> 10.0.1.124:80 dropped by fwmultik_process_f2p_cookie_inner Reason: fwmultik_f2p_cookie_outbound_and_routing failed;

[Masek]

Bad news, only a fraction of the "lost packets" are visible via "fw ctl zdebug + drop"

[the_rock]

See if below helps.

Andy

https://support.checkpoint.com/results/sk/sk167953

https://community.checkpoint.com/t5/Security-Gateways/dropped-by-fwmultik-process-f2p-cookie-inner-R...

[Masek · Accepted solution]

Lesson learned: When you use a a bridged interface (br0) in my case, you must not tcpdump on that interface but on the interface (in my case LAN4) which will receive the traffic. The packet went out.

Essential: do use "fw monitor -F", the you see all packets. Furthermore, the packet did not disappear but was becoming accelerated on the outgoing interface.

[the_rock]

Good job!