topic Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming in General Topics

FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

J_Saun — Fri, 03 Apr 2020 14:23:54 GMT

Trying to troubleshoot a connection. src=10.250.96.68, dest=10.129.3.191, port=445. Smartconsole logger shows it being permitted.

FW MONITOR shows the following:

[vs_0][fw_1] eth1:i[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth1:I[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth2:o[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth2:O[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth1:i[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth1:I[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth2:o[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth2:O[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272

When I do a tcpdump on eth1 I see the inbound packets from src-10.250.96.68 to dest-10.129.3.191, but a tcpdump on eth2 shows nothing.

I have tried with fwaccel off anf on. No difference.

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Vladimir — Fri, 03 Apr 2020 15:36:30 GMT

If this is a VSX cluster with a vSwitch connected to the eth1, you may have the egress/ingress traffic happening on a different cluster member.

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Timothy_Hall — Fri, 03 Apr 2020 15:37:57 GMT

Code version and Jumbo HFA?

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

J_Saun — Fri, 03 Apr 2020 15:43:32 GMT

This is a 5900 series appliance cluster (not VSX). R77.30 - Build 024

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Vladimir — Fri, 03 Apr 2020 15:53:13 GMT

ClusterXL or VRRP?

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Timothy_Hall — Fri, 03 Apr 2020 16:01:06 GMT

Try:

tcpdump -eP -i any -nnnl host 10.250.96.68 and host 10.129.3.191 and port 445

if that gives a syntax error try:

tcpdump -e -i any -nnnl host 10.250.96.68 and host 10.129.3.191 and port 445

Be aware that if the traffic is NATted, tcpdump may not display it correctly on your code version, see sk100194: TCPdump shows wrong IP addresses for NATed traffic when SecureXL is enabled and sk100071: "tcpdump" output does not show the NATed IP address correctly

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

J_Saun — Fri, 03 Apr 2020 16:39:04 GMT

We're using ClusterXL.

tcpdump results:

16:15:14.482890 eth1[in ]: 10.250.96.68.60206 > 10.129.3.192.445: S 3030511392:3030511392(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.482956 eth1[in ]: 10.250.96.68.60204 > 10.129.3.192.445: S 3875771840:3875771840(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483055 eth1[in ]: 10.250.96.68.60202 > 10.129.3.192.445: S 1196378115:1196378115(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483062 eth1[in ]: 10.250.96.68.60207 > 10.129.3.192.445: S 2233876772:2233876772(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483141 eth1[in ]: 10.250.96.68.60200 > 10.129.3.192.445: S 1879131292:1879131292(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483166 eth1[in ]: 10.250.96.68.60209 > 10.129.3.192.445: S 2463373099:2463373099(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>

Note: we're testing multiple hosts hence the 3.192 (not 3.191)

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Timothy_Hall — Sat, 04 Apr 2020 12:53:37 GMT

The traffic is actually leaving the firewall on the egress interface and connectivity is working correct? Not sure why tcpdump can't see the traffic leaving on the egress interface. I guess next step is to see if it is a problem with your filter or the libpcap portion. Try this:

tcpdump -eQ out -i (egress interface for your test traffic) -nnnl

Does tcpdump see anything at all leaving that interface outbound? If it does then it is some kind of problem with your filter, if not it has got to be some kind of bug in libpcap. Next:

tcpdump -e -i (egress interface for your test traffic) -nnnl -w capfile.out

(Run for a minute or so while initiating test traffic then CNTRL-C)

tcpdump -nnnl -r capfile.out (Make sure you have output)

tcpdump -nnnl -r capfile.out host 10.250.96.68 and host 10.129.3.191 and port 445

If the former command shows output, but the latter does not you have a problem with your filter.

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Vladimir — Sat, 04 Apr 2020 13:58:27 GMT

Please run same tcpdump on a different cluster member.

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

J_Saun — Tue, 07 Apr 2020 20:32:55 GMT

Figured it out. The gateway for the destination was a VIP (router pair). However it was not configured, only the single router nodes were. Changed my route to point to one of the routers and tcpdump started showing forward and return traffic. Is that a feature? I would at least expect to see the traffic leaving via tcpdump even tho the destination gateway wasn't available.

Thanks for all the help

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Timothy_Hall — Tue, 07 Apr 2020 21:01:12 GMT

Thanks for the followup. The traffic won't show up in tcpdump if a MAC address cannot be learned for the nonexistent gateway via ARP. There is no way to create a frame without a destination hardware address, and therefore the packet will never be transmitted. fw monitor showed it leaving via O out of INSPECT but it never went anywhere, tcpdump is plumbed in much lower in the networking stack than O.

Re: FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

J_Saun — Tue, 07 Apr 2020 21:05:53 GMT

Excellent. Thank you for the info. Much appreciated.