topic tcpdump and fw monitor missed packets in General Topics

tcpdump and fw monitor missed packets

Dawei_Ye — Tue, 04 Sep 2018 10:19:52 GMT

We are digging a issue with our application department.

Testing by our QA dept. the http connection could be a 5-6s latency occasionally.

So we did a packet capture.

the normal post and response:

the post that occurring latency as follows:

You could see the red column should be the POST request but the tcpdump shows "not captured"

and we also captured via fw monitor:

we can only see the POST request but no reponse:

Have you guys meeting this issues before?

Re: tcpdump and fw monitor missed packets

JozkoMrkvicka — Tue, 04 Sep 2018 11:15:28 GMT

Maybe because of SecureXL enabled ? Did you turn it off during debugs ?

Please check following thread before disable SecureXL:

To get the full output of fw monitor (and tcpdump) you should disable Secure XL with the command: "fwaccel off". You can re-enable it after debugs with the command: "fwaccel on". Another alternative is to disable SecureXL only for particular IPs, as is mentioned in the link above.

PS: You should blurry IPs in your screenshots.

Re: tcpdump and fw monitor missed packets

Dawei_Ye — Tue, 04 Sep 2018 11:25:17 GMT

Thank you ,Jozko.Blurred screenshots.

We disabled SecureXL.

Still the outputs as my screenshots.

Re: tcpdump and fw monitor missed packets

Vladimir — Tue, 04 Sep 2018 12:11:53 GMT

If this is a cluster of the gateways, I'd suggest using a span or mirror port on the switch(es) for definitive packet capture.

Have seen some asymmetrical weirdness a few times.

Re: tcpdump and fw monitor missed packets

Dawei_Ye — Tue, 04 Sep 2018 13:02:45 GMT

Hi Vladimir

Yes,our gateways are running clusterXL in Bridge mode.

You could see my second screenshots (captured on my WAN interface),actually ,the POST request is sent ,I think.But the tcpdump shows "TCP previous segment not captured".

Meanwhile,there is a normal output from our LAN interface ,but with latency.

So I don't think it is an asymmetrical problem.

Re: tcpdump and fw monitor missed packets

Vladimir — Tue, 04 Sep 2018 13:19:42 GMT

My point being is that you are looking at the traffic from L3 point of view only.

Incidentally, are you using vMAC on your clustered bridge?

And have you, perchance, added any other interfaces besides those in the bridge?

What kind of switches are on both sides of the bridge?

Thanks,

Vladimir

Re: tcpdump and fw monitor missed packets

JozkoMrkvicka — Tue, 04 Sep 2018 14:05:54 GMT

Can you please paste tcpdump and fw monitor command you have used ?

Re: tcpdump and fw monitor missed packets

Dawei_Ye — Wed, 05 Sep 2018 09:08:12 GMT

Hi Jozko,

these are commands for capture:

fw monitor -T -e "host(52.xx.xx.xx) or host(52.xx.xx.xx) and accept; "

tcpdump -e -w fw036-0904-wan.cap -i eth2-01 -nn host 52.xx.xx.xx or 52.xx.xx.xx -s 0
tcpdump -e -w fw036-0904-lan.cap -i eth2-02 -nn host 52.xx.xx.xx or 52.xx.xx.xx -s 0

52.xx.xx.xx are two servers used for test.

Re: tcpdump and fw monitor missed packets

Dawei_Ye — Wed, 05 Sep 2018 09:20:43 GMT

yes ,the customer have already check the issues with Application dept. and they have already captured the packets on server side ,there is no latency.

We didnt' use vMAC feature.

and besides brigde interfaces,there is only one Mgmt interface for updates and management.

Regards,

Dawei Ye

Re: tcpdump and fw monitor missed packets

HeikoAnkenbrand — Sun, 18 Nov 2018 20:02:25 GMT

SecureXL "fwaccel off" does not have to be disabled on R80.20 to run "fw monitor". This is good for performance, so "fw monitor" does not affect performance any more.

More see here: R80.x Performance Tuning and Debug Tips – fw monitor

Regards

Heiko

Re: tcpdump and fw monitor missed packets

Alan_Long — Mon, 09 Sep 2019 22:17:59 GMT

Did you ever get an answer to your question? We are seeing very similar to what you are getting

Re: tcpdump and fw monitor missed packets

Timothy_Hall — Sun, 15 Sep 2019 19:14:05 GMT

Could be indicative of frame loss at the NIC and/or NIC driver level, what does output of netstat -ni show?

Re: tcpdump and fw monitor missed packets

Alan_Long — Sun, 15 Sep 2019 20:10:49 GMT

We found the issue was due to a rule that should have no affect on the traffic flow. We disabled the rule and all is good. This same rule is on several of our other external clusters, and they have no issue at all. Support is looking into it now.

Re: tcpdump and fw monitor missed packets

Setu2 — Wed, 15 Mar 2023 11:41:53 GMT

Hi Alan,

did you get an anwser from support about this?or they creat any SK?

Re: tcpdump and fw monitor missed packets

_Val_ — Wed, 15 Mar 2023 11:53:21 GMT

@Setu2 this is a very old thread. With all supported versions today, fw monitor should show all the traffic, including fully accelerated packets. If you are still struggling, please open a new thread to discuss your issue.