Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
J_Saun
Contributor

FW MONITOR shows in IN, out Out, but tcpdump only shows incoming

Trying to troubleshoot a connection. src=10.250.96.68, dest=10.129.3.191, port=445. Smartconsole logger shows it being permitted.

FW MONITOR shows the following:

[vs_0][fw_1] eth1:i[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth1:I[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth2:o[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth2:O[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13271
[vs_0][fw_1] eth1:i[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth1:I[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth2:o[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272
[vs_0][fw_1] eth2:O[52]: 10.250.96.68 -> 10.129.3.191 (TCP) len=52 id=13272

When I do a tcpdump on eth1 I see the inbound packets from src-10.250.96.68 to dest-10.129.3.191, but a tcpdump on eth2 shows nothing.

I have tried with fwaccel off anf on. No difference.

0 Kudos
11 Replies
Vladimir
Champion
Champion

If this is a VSX cluster with a vSwitch connected to the eth1, you may have the egress/ingress traffic happening on a different cluster member.

0 Kudos
Timothy_Hall
Champion
Champion

Code version and Jumbo HFA?

Watch My 2023 CPX360 Speech Titled "Max Power
Reloaded: R81+ Gateway Performance Innovations"
0 Kudos
J_Saun
Contributor

This is a 5900 series appliance cluster (not VSX). R77.30 - Build 024

0 Kudos
Vladimir
Champion
Champion

ClusterXL or VRRP?

0 Kudos
Timothy_Hall
Champion
Champion

Try:

tcpdump -eP -i any -nnnl host 10.250.96.68 and host 10.129.3.191 and port 445

if that gives a syntax error try:

tcpdump -e -i any -nnnl host 10.250.96.68 and host 10.129.3.191 and port 445

Be aware that if the traffic is NATted, tcpdump may not display it correctly on your code version, see sk100194: TCPdump shows wrong IP addresses for NATed traffic when SecureXL is enabled and sk100071: "tcpdump" output does not show the NATed IP address correctly

 

Watch My 2023 CPX360 Speech Titled "Max Power
Reloaded: R81+ Gateway Performance Innovations"
0 Kudos
J_Saun
Contributor

We're using ClusterXL.

tcpdump results:

16:15:14.482890 eth1[in ]: 10.250.96.68.60206 > 10.129.3.192.445: S 3030511392:3030511392(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.482956 eth1[in ]: 10.250.96.68.60204 > 10.129.3.192.445: S 3875771840:3875771840(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483055 eth1[in ]: 10.250.96.68.60202 > 10.129.3.192.445: S 1196378115:1196378115(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483062 eth1[in ]: 10.250.96.68.60207 > 10.129.3.192.445: S 2233876772:2233876772(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483141 eth1[in ]: 10.250.96.68.60200 > 10.129.3.192.445: S 1879131292:1879131292(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>
16:15:14.483166 eth1[in ]: 10.250.96.68.60209 > 10.129.3.192.445: S 2463373099:2463373099(0) win 8192 <mss 1334,nop,wscale 8,nop,nop,sackOK>

 

Note: we're testing multiple hosts hence the 3.192 (not 3.191)

0 Kudos
Timothy_Hall
Champion
Champion

The traffic is actually leaving the firewall on the egress interface and connectivity is working correct?  Not sure why tcpdump can't see the traffic leaving on the egress interface.  I guess next step is to see if it is a problem with your filter or the libpcap portion.  Try this:

tcpdump -eQ out -i (egress interface for your test traffic) -nnnl

Does tcpdump see anything at all leaving that interface outbound?  If it does then it is some kind of problem with your filter, if not it has got to be some kind of bug in libpcap.  Next:

tcpdump -e -i (egress interface for your test traffic) -nnnl -w capfile.out

(Run for a minute or so while initiating test traffic then CNTRL-C)

tcpdump -nnnl -r capfile.out (Make sure you have output)

tcpdump -nnnl -r capfile.out host 10.250.96.68 and host 10.129.3.191 and port 445

If the former command shows output, but the latter does not you have a problem with your filter.

Watch My 2023 CPX360 Speech Titled "Max Power
Reloaded: R81+ Gateway Performance Innovations"
0 Kudos
Vladimir
Champion
Champion

Please run same tcpdump on a different cluster member.

0 Kudos
J_Saun
Contributor

Figured it out. The gateway for the destination was a VIP (router pair). However it was not configured, only the single router nodes were. Changed my route to point to one of the routers and tcpdump started showing forward and return traffic. Is that a feature? I would at least expect to see the traffic leaving via tcpdump even tho the destination gateway wasn't available.

Thanks for all the help

0 Kudos
Timothy_Hall
Champion
Champion

Thanks for the followup.  The traffic won't show up in tcpdump if a MAC address cannot be learned for the nonexistent gateway via ARP.  There is no way to create a frame without a destination hardware address, and therefore the packet will never be transmitted.  fw monitor showed it leaving via O out of INSPECT but it never went anywhere, tcpdump is plumbed in much lower in the networking stack than O.

Watch My 2023 CPX360 Speech Titled "Max Power
Reloaded: R81+ Gateway Performance Innovations"
0 Kudos
J_Saun
Contributor

Excellent. Thank you for the info. Much appreciated.

0 Kudos