FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

[J_Saun]

Trying to troubleshoot a connection. src=10.250.96.68, dest=10.129.3.191, port=445. Smartconsole logger shows it being permitted.

FW MONITOR shows the following:

[vs_0][fw_1] eth1:i[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth1:I[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth2:o[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth2:O[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth1:i[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth1:I[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth2:o[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth2:O[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272

When I do a tcpdump on eth1 I see the inbound packets from src-10.250.96.68 to dest-10.129.3.191, but a tcpdump on eth2 shows nothing.

I have tried with fwaccel off anf on. No difference.

[Vladimir]

If this is a VSX cluster with a vSwitch connected to the eth1, you may have the egress/ingress traffic happening on a different cluster member.

[Timothy_Hall]

Code version and Jumbo HFA?

[J_Saun]

This is a 5900 series appliance cluster (not VSX). R77.30 - Build 024

[Vladimir]

ClusterXL or VRRP?

[Timothy_Hall]

Try:

tcpdump -eP -i any -nnnl host 10.250.96.68 and host 10.129.3.191 and port 445

if that gives a syntax error try:

tcpdump -e -i any -nnnl host 10.250.96.68 and host 10.129.3.191 and port 445

Be aware that if the traffic is NATted, tcpdump may not display it correctly on your code version, see sk100194: TCPdump shows wrong IP addresses for NATed traffic when SecureXL is enabled and sk100071: "tcpdump" output does not show the NATed IP address correctly

[J_Saun]

We're using ClusterXL.

tcpdump results:

16:15:14.482890 eth1[in ]: 10.250.96.68.60206 > 10.129.3.192.445: S 3030511392:3030511392(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.482956 eth1[in ]: 10.250.96.68.60204 > 10.129.3.192.445: S 3875771840:3875771840(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483055 eth1[in ]: 10.250.96.68.60202 > 10.129.3.192.445: S 1196378115:1196378115(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483062 eth1[in ]: 10.250.96.68.60207 > 10.129.3.192.445: S 2233876772:2233876772(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483141 eth1[in ]: 10.250.96.68.60200 > 10.129.3.192.445: S 1879131292:1879131292(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483166 eth1[in ]: 10.250.96.68.60209 > 10.129.3.192.445: S 2463373099:2463373099(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>

Note: we're testing multiple hosts hence the 3.192 (not 3.191)

[Timothy_Hall]

The traffic is actually leaving the firewall on the egress interface and connectivity is working correct?  Not sure why tcpdump can't see the traffic leaving on the egress interface.  I guess next step is to see if it is a problem with your filter or the libpcap portion.  Try this:

tcpdump -eQ out -i (egress interface for your test traffic) -nnnl

Does tcpdump see anything at all leaving that interface outbound?  If it does then it is some kind of problem with your filter, if not it has got to be some kind of bug in libpcap.  Next:

tcpdump -e -i (egress interface for your test traffic) -nnnl -w capfile.out

(Run for a minute or so while initiating test traffic then CNTRL-C)

tcpdump -nnnl -r capfile.out (Make sure you have output)

tcpdump -nnnl -r capfile.out host 10.250.96.68 and host 10.129.3.191 and port 445

If the former command shows output, but the latter does not you have a problem with your filter.

[Vladimir]

Please run same tcpdump on a different cluster member.

[J_Saun]

Figured it out. The gateway for the destination was a VIP (router pair). However it was not configured, only the single router nodes were. Changed my route to point to one of the routers and tcpdump started showing forward and return traffic. Is that a feature? I would at least expect to see the traffic leaving via tcpdump even tho the destination gateway wasn't available.

Thanks for all the help

[Timothy_Hall]

Thanks for the followup.  The traffic won't show up in tcpdump if a MAC address cannot be learned for the nonexistent gateway via ARP.  There is no way to create a frame without a destination hardware address, and therefore the packet will never be transmitted.  fw monitor showed it leaving via O out of INSPECT but it never went anywhere, tcpdump is plumbed in much lower in the networking stack than O.

[J_Saun]

Excellent. Thank you for the info. Much appreciated.