R80.40 Bridge Mode - Unknown Unicast traffic - Source Mac Issue

[leonarit]

Hello,

I would like to check with the community if anyone has an idea to how the following behaviour may be happening.

We have an L2 link to another site and would like to take advantage of two 13500(R80.40) that were recently retired,  to be able gain some visibility and control (L3/L4) of the traffic that passes through the L2 link.

So, the two 13500 were reinstalled from the base image R80.20 and upgraded to R80.40 with JHF197.

Current setup:

Both fws are configured as standalone.

FW1 Interfaces:
BOND1(2 LINKS LACP + HASH L3+L4)
BRIDGE1 (BOND1(LAN) –ETH3-03(WAN-L2-LINK)
BOND1.1156 – Subnet /30 used only for fw management.
ETH3-03.1156 – Subnet /30 used only for fw management.

FW2 Interfaces:
BOND1(2 LINKS LACP - HASH L3+L4)
BRIDGE1 (BOND1(LAN) –ETH3-03(WAN-L2-LINK)
ETH3-03.1156 – Subnet /30 used only for fw management.

Blades(FW1/FW2):
– Firewall Blade only

Policy(FW1/FW2)
Access
Rule1 - Permit Any Any log (Only for tests)

We did some initial tests and both the L2 and L3 traffic is being handled correctly between the sites 1/2.

Server 1A in SITE1 sends some unicast tagged traffic destined to SITE2, the FW1 and FW2 inspects the traffic in L2 mode, and the server 2 in SITE2 receives the tagged traffic. Ther other was works also.

Now the real problem, the Servers 1A in SITE1 needs to send some packets to a virtual mac (02:bf:ac:10:0a:fe) used by the NBL function present in some windows systems (Servers 1B) that resides in SITE1,  this virtual mac is used only to receive traffic from other servers present in SITE1, currently the NBL is configured in unicast mode, so every frame send to the NBL  virtual mac is considered unknown unicast traffic, SW1 from SITE1 floods this frames to all the ports that belong to that vlan, the SW1 ports connected to the BOND1 have this vlan allowed.

Although this traffic doesn’t need to pass FW1 when it reaches FW1 and it tries to inspect the traffic, after the exchange of some packets when the TCP connection is normally closed by the client(TCP RST), sometimes the firewall loops back the (TCP RST) packet to the SW1 through the BOND1 interface. Because of the previous behaviour SW1 learns the virtual NBL mac from the FW1 and sees the virtual NBL nbl present on the BOND1 interface, after that all the traffic destined to the virtual NBL mac gets send to the FW1 incorrectly.

The easy fix is to disable this vlan on the BOND1 and for now we had to do that, if not the servers weren't able to talk to each other trough the virtual nbl mac.

I need to understand whats causing this issue because in the future we will need to be able to pass this traffic trough the L2 WAN.

I’ve disabled securexl (fwacl off) and collected some traces from the firewall with “tcpdump -I bond1 -en ether source ‘virtual nbl mac’” and the only frames I see with source mac of virtual nbl mac are TCP RST packets.

05:29:47.343744 02:bf:ac:10:0a:fe > 00:50:56:a4:1f:04, ethertype 802.1Q (0x8100), length 58: vlan 2600, p 0, ethertype IPv4, 172.20.4.2.56526 > 172.16.10.36.88: Flags [R], seq 89268862, win 0, length 0
05:29:47.344064 02:bf:ac:10:0a:fe > 00:50:56:a4:1f:04, ethertype 802.1Q (0x8100), length 58: vlan 2600, p 0, ethertype IPv4, 172.20.4.2.56526 > 172.16.10.36.88: Flags [R], seq 89268863, win 0, length 0
05:29:47.345093 02:bf:ac:10:0a:fe > 00:50:56:a4:1f:04, ethertype 802.1Q (0x8100), length 58: vlan 2600, p 0, ethertype IPv4, 172.20.4.2.56527 > 172.16.10.36.88: Flags [R], seq 609578074, win 0, length 0
05:29:47.346406 02:bf:ac:10:0a:fe > 00:50:56:a4:1f:04, ethertype 802.1Q (0x8100), length 58: vlan 2600, p 0, ethertype IPv4, 172.20.4.2.56527 > 172.16.10.36.88: Flags [R], seq 609578074, win 0, length 0
05:29:47.346446 02:bf:ac:10:0a:fe > 00:50:56:a4:1f:04, ethertype 802.1Q (0x8100), length 58: vlan 2600, p 0, ethertype IPv4, 172.20.4.2.56527 > 172.16.10.36.88: Flags [R], seq 609578074, win 0, length 0
05:29:47.346905 02:bf:ac:10:0a:fe > 00:50:56:a4:1f:04, ethertype 802.1Q (0x8100), length 58: vlan 2600, p 0, ethertype IPv4, 172.20.4.2.56527 > 172.16.10.36.88: Flags [R], seq 609578075, win 0, length 0

I thought that maybe the firewall was generating this tcp packets but the "fw_rst_expired_conn” is disabled.

If I remove the bond0 from the br1 the problem doesn’t happen, I’ve also reviewed the traffic from FW2 and I don’t see this TCP RST packets being send to FW1.

I need to do some additional tests to be able to be sure if the bridge of FW1 somehow is looping back the TCP RST packets to SW1 or generating new tcp packets with the source mac 02:bf:ac:10:0a:fe.

I’ve tried to do a fw monitor “fw monitor -T -p all -e "host(172.20.4.2) and host(172.16.10.36), accept;”  but I don’t see the TCP RST packets I’ve captured with tcdump. I will try again to do a fw monitor with the -F flag.

Doesn’t anyone in the community had any kind of similar deployment and face some similar issues?

Regards.

[Chris_Atkinson]

Most every Microsoft NLB environment I saw  ended up with static cam table entries in the switches because having that traffic smashing about was undesirable irrespective of what your attempting to achieve with the firewalls.

Please note 13500 appliances are End of Support and a similar milestone for R80.40 is also approaching.

[the_rock]

This definitely might be worth TAC case. 

You could also try fw monitor -F syntax, so say, just for the context, if src is 1.1.1.1 and dst is 2.2.2.2 and port 443, it would look something like below:

fw monitor -F "1.1.1.1,0,2.2.2.2,443,0" -F "2.2.2.2,0,1.1.1.1,443,0"

Idea is this "srcip,srcport,dstip,dstport,protocol"

Andy

[leonarit · Accepted solution]

Finally, I was able to identify the root cause.

It seems the firewall is dealing with a concurrency issue and for some traffic flows the firewall sends an RST packet to clear the inconsistent connection.

The issue happened in this flow: Client 172.20.4.2(TCP/57490) -> Server 172.16.10.36(TCP/135), both the client and server were using Windows LNB in unicast mode.

While the fw was still processing the first SYN packet, simultaneously received an SYN+ACK for the same connection, because the processing of the first SYN wasn’t finished the SYN+ACK was discarded. This behavior is visible In the kernel fw debug and in the fw monitor.

After discarding the SYN+ACK packet the fw kept trying to process the following packets of the connection, after receiving an PUSH+ACK and because it was still waiting for the ACK to the first SYN, it sends a RST packet to both parties to clear the connection. Because this inspection is being done in L2 mode, the mac address used by the firewall is the MAC address of the client (Unicast NBL mac) thus allowing the switch to wrongly learn the mac address of the NLB cluster in the switch port connected to the firewall.

SYN-ACK discard

[-- Stateful VM inbound: Entering (1689918324) --];
@;27479832;21Jul2023  6:45:24.620991;[cpu_24];[fw4_14];fwconn_key_set_links_inbound: create link cls_o <dir 1, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> -> <dir 0, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6>(0x5);
@;27479832;21Jul2023  6:45:24.620996;[cpu_4];[fw4_23];Before VM: <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> (len=52) TCP flags=0x12 (SYN-ACK), seq=2770938187, ack=496573280, data end=2770938188 (ifn=19) (first seen) ;
@;27479832;21Jul2023  6:45:24.620999;[cpu_24];[fw4_14];fwconn_key_set_link (normal): conn <dir 1, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> added to dispatcher with ppack_id=-1 (qid=-1), member_id=-1, flags=1;
@;27479832;21Jul2023  6:45:24.621001;[cpu_4];[fw4_23];fwconn_lookup_cache: conn <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6>;
@;27479832;21Jul2023  6:45:24.621004;[cpu_4];[fw4_23];fwconn_lookup_cache: not found in connections table. nvals = 0;
@;27479832;21Jul2023  6:45:24.621009;[cpu_24];[fw4_14];fwconn_key_set_links_inbound: create link srs_o <dir 1, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6> -> <dir 0, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6>(0x2);
@;27479832;21Jul2023  6:45:24.621011;[cpu_4];[fw4_23];fw_first_packet_state_checks: Executing accept_non_syn inspect scripts (internal id=216, user id=-1);
@;27479832;21Jul2023  6:45:24.621015;[cpu_4];[fw4_23];fwconnoxid_msg_get_cliconn: No connoxid message on <dir 1, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6>;
@;27479832;21Jul2023  6:45:24.621016;[cpu_24];[fw4_14];fwconn_key_set_link (normal): conn <dir 1, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6> added to dispatcher with ppack_id=-1 (qid=-1), member_id=-1, flags=1;
@;27479832;21Jul2023  6:45:24.621018;[cpu_24];[fw4_14];fwconn_key_set_links_inbound: VM-side links created successfuly;
@;27479832;21Jul2023  6:45:24.621020;[cpu_24];[fw4_14];fwconn_key_init_links: connection completed, unset PARTIAL_SETUP bit;
@;27479832;21Jul2023  6:45:24.621024;[cpu_24];[fw4_14];fwconnoxid_msg_get_cliconn: No connoxid message on <dir 1, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6>;
@;27479832;21Jul2023  6:45:24.621030;[cpu_4];[fw4_23];fwconn_lookup_other_ex__legacy__: conn <dir 0, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6>
not found in connections table;
@;27479832;21Jul2023  6:45:24.621033;[cpu_24];[fw4_14];fwconn_key_init_links: Creating links (outbound). One way links=0, Replies from any=0;
@;27479832;21Jul2023  6:45:24.621035;[cpu_4];[fw4_23];fwconn_key_lookup_ex: conn <dir 0, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6> not found in connections table.;
@;27479832;21Jul2023  6:45:24.621038;[cpu_24];[fw4_14];fwconn_key_set_links_outbound: create link srs_i <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> -> <dir 0, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6>(0x6);
@;27479832;21Jul2023  6:45:24.621040;[cpu_4];[fw4_23];FW-1: fw_log_tcp_out_of_state: reason First packet isn't SYN. th_flags 0x12;
@;27479832;21Jul2023  6:45:24.621043;[cpu_4];[fw4_23];fw_first_packet_state_checks: Dropping out of state TCP packet;
@;27479832;21Jul2023  6:45:24.621046;[cpu_24];[fw4_14];fwconn_key_set_link (normal): conn <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> added to dispatcher with ppack_id=-1 (qid=-1), member_id=-1, flags=1;
@;27479832;21Jul2023  6:45:24.621046;[cpu_4];[fw4_23];fw_first_packet_state_checks: chain->dir=0, dir=0, th_flags=0x12, src=172.16.10.36, dst=172.20.4.2;
@;27479832;21Jul2023  6:45:24.621051;[cpu_24];[fw4_14];fwconn_key_set_links_outbound: server-side links created successfuly;
@;27479832;21Jul2023  6:45:24.621052;[cpu_24];[fw4_14];fw_first_packet_outbound_init: Setting DONE_OUTBOUND_INIT flag;
@;27479832;21Jul2023  6:45:24.621053;[cpu_4];[fw4_23];fw_log_drop_ex: Packet proto=6 172.16.10.36:135 -> 172.20.4.2:57490 dropped by fw_first_packet_state_checks Reason: First packet isn't SYN;
@;27479832;21Jul2023  6:45:24.621056;[cpu_4];[fw4_23];fw_handle_first_packet: first packet state violation (action=DROP);
@;27479832;21Jul2023  6:45:24.621061;[cpu_4];[fw4_23];After  VM: <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> (len=52) TCP flags=0x12 (SYN-ACK), seq=2770938187, ack=496573280, data end=2770938188 ;
@;27479832;21Jul2023  6:45:24.621061;[cpu_24];[fw4_14];After  VM: <dir 0, 172.20.4.2:57490 -> 172.16.10.36:135 IPP 6> (len=52) TCP flags=0x2 (SYN), seq=496573279, ack=0, data end=496573280 ;
@;27479832;21Jul2023  6:45:24.621064;[cpu_24];[fw4_14];VM Final action=ACCEPT;
@;27479832;21Jul2023  6:45:24.621065;[cpu_4];[fw4_23];VM Final action=DROP;
@;27479832;21Jul2023  6:45:24.621066;[cpu_4];[fw4_23]; -----  Stateful VM inbound Completed -----

Connection reset:

@;27479832;21Jul2023  6:45:24.621942;[cpu_24];[fw4_14];
	   [-- Stateful VM inbound: Entering (1689918324) --];
	@;27479832;21Jul2023  6:45:24.621945;[cpu_24];[fw4_14];Before VM: <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> (len=148) TCP flags=0x18 (PUSH-ACK), seq=2770938188, ack=496573440, data end=2770938296 (ifn=19) (first seen) ;
	@;27479832;21Jul2023  6:45:24.621947;[cpu_24];[fw4_14];fwconn_lookup_cache: conn <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6>;
	@;27479832;21Jul2023  6:45:24.621954;[cpu_24];[fw4_14];<10001,40046080,1,e10,64,UUID: 64ba1b74-0001-0001-bd-90-8d-ee-fa-dd-ec-d8, 13,ffffffff,ffffffff,11,2007800,7c800,80000084,OPQS:[ffffc900b9e11648,ffffc900b950fc90,0,ffffc900b868bab0,0,0,ffffc900c0e7dad8,0,0,0,ffffc900bad1be88,0,0,0,0,0,ffffc900bc39aee0,0,0,0],0,0,0,0,0,0,0,0,0,0,0,24855800,0,0>
	found in connections table dir=2 keytype=806;
	@;27479832;21Jul2023  6:45:24.621956;[cpu_24];[fw4_14];fw_tcp_state_verification: Received ACK as response to SYN packet. Sending RST on behalf of the client - tcp flags=18, ctype=10001, cflags=40046080;
	@;27479832;21Jul2023  6:45:24.621959;[cpu_24];[fw4_14];fw_log_drop_ex: Packet proto=6 172.16.10.36:135 -> 172.20.4.2:57490 dropped by fw_tcp_state_verification Reason: ACK response to SYN. Sending RST on behalf of the client;
	@;27479832;21Jul2023  6:45:24.621960;[cpu_24];[fw4_14];fw_conn_inspect: fw_tcp_state_verification returned REJECT;
	@;27479832;21Jul2023  6:45:24.621962;[cpu_24];[fw4_14];After  VM: <dir 0, 172.16.10.36:135 -> 172.20.4.2:57490 IPP 6> (len=148) TCP flags=0x18 (PUSH-ACK), seq=2770938188, ack=496573440, data end=2770938296 ;
	@;27479832;21Jul2023  6:45:24.621963;[cpu_24];[fw4_14];VM Final action=REJECT;
	@;27479832;21Jul2023  6:45:24.621964;[cpu_24];[fw4_14]; -----  Stateful VM inbound Completed -----

FW monitor

TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.620576 bond1:i10 (Stateless verifications (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.620587 bond1:i11 (fw multik misc proto forwarding)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.620599 bond1:i12 (fw VM inbound )[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_23] 21Jul2023  6:45:24.620950 bond1:i9 (IP Options Strip (in))[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=52 id=11729
TCP: 135 -> 57490 .S..A. seq=a529294b ack=1d991b60
[vs_0][fw_23] 21Jul2023  6:45:24.620963 bond1:i10 (Stateless verifications (in))[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=52 id=11729
TCP: 135 -> 57490 .S..A. seq=a529294b ack=1d991b60
[vs_0][fw_23] 21Jul2023  6:45:24.620976 bond1:i11 (fw multik misc proto forwarding)[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=52 id=11729
TCP: 135 -> 57490 .S..A. seq=a529294b ack=1d991b60
[vs_0][fw_23] 21Jul2023  6:45:24.620989 bond1:i12 (fw VM inbound )[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=52 id=11729
TCP: 135 -> 57490 .S..A. seq=a529294b ack=1d991b60
[vs_0][fw_14] 21Jul2023  6:45:24.621084 bond1:I13 (fw SCV inbound)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621097 bond1:I14 (fw offload inbound)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621125 bond1:I15 (fw post VM inbound )[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621170 bond1:I16 (passive streaming (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621189 bond1:I17 (TCP streaming (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621201 bond1:I18 (IP Options Restore (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621211 bond1:I19 (Chain End)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621243 eth3-03:o0 (IP Options Strip (out))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621262 eth3-03:o1 (TCP streaming (out))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621281 eth3-03:o2 (passive streaming (out))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621290 eth3-03:o3 (Stateless verifications (out))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621300 eth3-03:o4 (fw VM outbound)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621323 eth3-03:O5 (fw post VM outbound )[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621345 eth3-03:O6 (TCP streaming post VM)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621355 eth3-03:O7 (IP Options Restore (out))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621365 eth3-03:O13 (Chain End)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=52 id=10946
TCP: 57490 -> 135 .S.... seq=1d991b5f ack=00000000
[vs_0][fw_14] 21Jul2023  6:45:24.621407 bond1:i9 (IP Options Strip (in))[40]: 172.20.4.2 -> 172.16.10.36 (TCP) len=40 id=10947
TCP: 57490 -> 135 ....A. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621439 bond1:i10 (Stateless verifications (in))[40]: 172.20.4.2 -> 172.16.10.36 (TCP) len=40 id=10947
TCP: 57490 -> 135 ....A. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621462 bond1:i11 (fw multik misc proto forwarding)[40]: 172.20.4.2 -> 172.16.10.36 (TCP) len=40 id=10947
TCP: 57490 -> 135 ....A. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621481 bond1:i12 (fw VM inbound )[40]: 172.20.4.2 -> 172.16.10.36 (TCP) len=40 id=10947
TCP: 57490 -> 135 ....A. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621510 bond1:I13 (fw SCV inbound)[40]: 172.20.4.2 -> 172.16.10.36 (TCP) len=40 id=10947
TCP: 57490 -> 135 ....A. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621520 bond1:I14 (fw offload inbound)[40]: 172.20.4.2 -> 172.16.10.36 (TCP) len=40 id=10947
TCP: 57490 -> 135 ....A. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621530 bond1:I15 (fw post VM inbound )[40]: 172.20.4.2 -> 172.16.10.36 (TCP) len=40 id=10947
TCP: 57490 -> 135 ....A. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621606 bond1:i9 (IP Options Strip (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=200 id=10948
TCP: 57490 -> 135 ...PA. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621631 bond1:i10 (Stateless verifications (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=200 id=10948
TCP: 57490 -> 135 ...PA. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621651 bond1:i11 (fw multik misc proto forwarding)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=200 id=10948
TCP: 57490 -> 135 ...PA. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621670 bond1:i12 (fw VM inbound )[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=200 id=10948
TCP: 57490 -> 135 ...PA. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621697 bond1:I13 (fw SCV inbound)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=200 id=10948
TCP: 57490 -> 135 ...PA. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621707 bond1:I14 (fw offload inbound)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=200 id=10948
TCP: 57490 -> 135 ...PA. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621717 bond1:I15 (fw post VM inbound )[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=200 id=10948
TCP: 57490 -> 135 ...PA. seq=1d991b60 ack=a529294c
[vs_0][fw_14] 21Jul2023  6:45:24.621884 bond1:i9 (IP Options Strip (in))[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=148 id=11730
TCP: 135 -> 57490 ...PA. seq=a529294c ack=1d991c00
[vs_0][fw_14] 21Jul2023  6:45:24.621903 bond1:i10 (Stateless verifications (in))[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=148 id=11730
TCP: 135 -> 57490 ...PA. seq=a529294c ack=1d991c00
[vs_0][fw_14] 21Jul2023  6:45:24.621922 bond1:i11 (fw multik misc proto forwarding)[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=148 id=11730
TCP: 135 -> 57490 ...PA. seq=a529294c ack=1d991c00
[vs_0][fw_14] 21Jul2023  6:45:24.621941 bond1:i12 (fw VM inbound )[44]: 172.16.10.36 -> 172.20.4.2 (TCP) len=148 id=11730
TCP: 135 -> 57490 ...PA. seq=a529294c ack=1d991c00
[vs_0][fw_14] 21Jul2023  6:45:24.622144 bond1:i9 (IP Options Strip (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=208 id=10949
TCP: 57490 -> 135 ...PA. seq=1d991c00 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622173 bond1:i10 (Stateless verifications (in))[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=208 id=10949
TCP: 57490 -> 135 ...PA. seq=1d991c00 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622198 bond1:i11 (fw multik misc proto forwarding)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=208 id=10949
TCP: 57490 -> 135 ...PA. seq=1d991c00 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622218 bond1:i12 (fw VM inbound )[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=208 id=10949
TCP: 57490 -> 135 ...PA. seq=1d991c00 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622246 bond1:I13 (fw SCV inbound)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=208 id=10949
TCP: 57490 -> 135 ...PA. seq=1d991c00 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622256 bond1:I14 (fw offload inbound)[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=208 id=10949
TCP: 57490 -> 135 ...PA. seq=1d991c00 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622265 bond1:I15 (fw post VM inbound )[44]: 172.20.4.2 -> 172.16.10.36 (TCP) len=208 id=10949
TCP: 57490 -> 135 ...PA. seq=1d991c00 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622326 bond1:i9 (IP Options Strip (in))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622345 bond1:i10 (Stateless verifications (in))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622363 bond1:i11 (fw multik misc proto forwarding)[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622381 bond1:i12 (fw VM inbound )[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622412 bond1:I13 (fw SCV inbound)[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622421 bond1:I14 (fw offload inbound)[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622431 bond1:I15 (fw post VM inbound )[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622462 bond1:I16 (passive streaming (in))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622473 bond1:I17 (TCP streaming (in))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622483 bond1:I18 (IP Options Restore (in))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622493 bond1:I19 (Chain End)[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622517 eth3-03:o0 (IP Options Strip (out))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622535 eth3-03:o1 (TCP streaming (out))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622553 eth3-03:o2 (passive streaming (out))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622564 eth3-03:o3 (Stateless verifications (out))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622574 eth3-03:o4 (fw VM outbound)[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622593 eth3-03:O5 (fw post VM outbound )[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622614 eth3-03:O6 (TCP streaming post VM)[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622624 eth3-03:O7 (IP Options Restore (out))[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490..R... seq=a52929b8 ack=a52929b8
[vs_0][fw_14] 21Jul2023  6:45:24.622633 eth3-03:O13 (Chain End)[40]: 172.16.10.36 -> 172.20.4.2 (TCP) len=40 id=11731
TCP: 135 -> 57490 ..R... seq=a52929b8 ack=a52929b8

The correct fix for this issue should be to configure the NLB in multicast igmp mode instead of unicast, as an emergency measure one could try to disable the stateful inspection associated with these servers as mentioned in this topic.