Packet corruption by firewall

[Hugo_vd_Kooij]

I raised the issue with TAC. But I was wondering if someone has ever observed that sessions fail due to the gateway mishandling the packet.

I observice this in a pcap gathered with fw monitor.

On the (i) stage I have a Normal SYN packet for the new session inluding a MSS value of 1460.

On the (I) stage the SYN flag is gone and an ACK flag appeared out of thin air.

Everything else looks the same. A redacted screenshot is attached.

The mishandling is consistent for this particular session. The next connection from the same client to the same host hapens without incident.

It happens on a minority of the  sessions.

[Timothy_Hall]

sk24960: "Smart Connection Reuse" feature modifies some SYN packets

[Hugo_vd_Kooij]

I was checking this and found the setting:

[Expert@fw01:0]# fw ctl get int fwconn_smart_conn_reuse
fwconn_smart_conn_reuse = 1

I will make a change and see what happens.

[the_rock]

Definitely could be the issue. I recall spending whole night just before covid-19 hit at customer's site when they upgraded from R80.20 to R80.30 with TAC T3 and escalation and we ended up finding this setting ourselves and once turned off, it fixed the problem. It was solved later with jumbo that came out...

Cheers,

Andy

[Timothy_Hall]

Do you have SYN Defender enabled?  This issue was just fixed by the most recent Jumbo HFAs:

PRJ-49379, PRHF-30056

SecureXL

SYN Defender may not correctly handle reused connections.

[Hugo_vd_Kooij]

Will look into that today. As I followed sk24960: "Smart Connection Reuse" feature modifies some SYN packets yesterday and I still don't get dropped packets in the logs. Not even in the zdebug output.

Someone raised the default idle timmer from 1 hour to 4 hours on this firewall. And digging into the connection table I could find connections that were idle for almost 3 hours. And if I read sk65133: Connections Table Format correctly it seems these sessins have only seen thre FIN packets from the server and not from the client.

So the firewall is being stateful and keeps them open waiting for the other FIN.

May do some serious packet capturing over the course of the day to find the reason for this.

(2 hosting parties, one of which is Azure, a VPN tunnel, a MPLS link, .... only a dozen things that could go wrong here 😉 )

[Hugo_vd_Kooij]

Oh and we have another weird thing on the standby member that drops HTTPS a lot on IPS stuff like maximum header length exceeded. So instaling a Jumbo Hotfix is not on the table as an option at the moment.