tcpdump and fw monitor missed packets

[Dawei_Ye]

We are digging a issue with our application department.

Testing by our QA dept. the http connection could be a 5-6s latency occasionally.

So we did a packet capture.

the normal post and response:

the post that occurring latency as follows:

You could see the red column should be the POST request but the tcpdump shows "not captured"

and we also captured via fw monitor:

we can only see the POST request but no reponse:

Have you guys meeting this issues before?

[JozkoMrkvicka]

Maybe because of SecureXL enabled ? Did you turn it off during debugs ?

Please check following thread before disable SecureXL:

To get the full output of fw monitor (and tcpdump) you should disable Secure XL with the command: "fwaccel off". You can re-enable it after debugs with the command: "fwaccel on". Another alternative is to disable SecureXL only for particular IPs, as is mentioned in the link above.

PS: You should blurry IPs in your screenshots.

[Dawei_Ye]

Thank you ,Jozko.Blurred screenshots.

We disabled SecureXL.

Still the outputs as my screenshots.

[Vladimir]

If this is a cluster of the gateways, I'd suggest using a span or mirror port on the switch(es) for definitive packet capture.

Have seen some asymmetrical weirdness a few times. 

[Dawei_Ye]

Hi Vladimir

Yes,our gateways are running clusterXL in Bridge mode.

You could see my second screenshots (captured on my WAN interface),actually ,the POST request is sent ,I think.But the tcpdump shows "TCP previous segment not captured".

Meanwhile,there is a normal output from our LAN interface ,but with latency.

So I don't think it is an asymmetrical problem.

[Vladimir]

My point being is that you are looking at the traffic from L3 point of view only.

Incidentally, are you using vMAC on your clustered bridge?

And have you, perchance, added any other interfaces besides those in the bridge?

What kind of switches are on both sides of the bridge?

Thanks,

Vladimir

[Dawei_Ye]

yes ,the customer have already check the issues with Application dept. and they have already captured the packets on server side ,there is no latency.

We didnt' use vMAC feature.

and besides brigde interfaces,there is only one Mgmt interface for updates and management.

Regards,

Dawei Ye

[JozkoMrkvicka]

Can you please paste tcpdump and fw monitor command you have used ?

[Dawei_Ye]

Hi Jozko,

these are commands for capture:

fw monitor -T -e "host(52.xx.xx.xx) or host(52.xx.xx.xx) and accept; "

tcpdump -e -w fw036-0904-wan.cap -i eth2-01 -nn host 52.xx.xx.xx or 52.xx.xx.xx -s 0
tcpdump -e -w fw036-0904-lan.cap -i eth2-02 -nn host 52.xx.xx.xx or 52.xx.xx.xx -s 0

52.xx.xx.xx are two servers used for test.

[HeikoAnkenbrand]

SecureXL "fwaccel off" does not have to be disabled on R80.20 to run "fw monitor". This is good for performance, so "fw monitor" does not affect performance any more.

More see here: R80.x Performance Tuning and Debug Tips – fw monitor 

Regards

Heiko

[Alan_Long]

Did you ever get an answer to your question? We are seeing very similar to what you are getting

[Timothy_Hall]

Could be indicative of frame loss at the NIC and/or NIC driver level, what does output of netstat -ni show?

[Alan_Long]

We found the issue was due to a rule that should have no affect on the traffic flow. We disabled the rule and all is good. This same rule is on several of our other external clusters, and they have no issue at all. Support is looking into it now.

[Setu2]

Hi Alan,

did you get an anwser from support about this?or they creat any SK?

[_Val_]

@Setu2 this is a very old thread. With all supported versions today, fw monitor should show all the traffic, including fully accelerated packets. If you are still struggling, please open a new thread to discuss your issue.