topic Re: Packet corruption by firewall in Firewall and Security Management

Packet corruption by firewall

Hugo_vd_Kooij — Fri, 01 Dec 2023 14:19:22 GMT

I raised the issue with TAC. But I was wondering if someone has ever observed that sessions fail due to the gateway mishandling the packet.

I observice this in a pcap gathered with fw monitor.

On the (i) stage I have a Normal SYN packet for the new session inluding a MSS value of 1460.

On the (I) stage the SYN flag is gone and an ACK flag appeared out of thin air.

Everything else looks the same. A redacted screenshot is attached.

The mishandling is consistent for this particular session. The next connection from the same client to the same host hapens without incident.

It happens on a minority of the sessions.

Re: Packet corruption by firewall

Timothy_Hall — Fri, 01 Dec 2023 14:46:08 GMT

sk24960: "Smart Connection Reuse" feature modifies some SYN packets

Re: Packet corruption by firewall

Hugo_vd_Kooij — Mon, 04 Dec 2023 12:51:56 GMT

I was checking this and found the setting:

[Expert@fw01:0]# fw ctl get int fwconn_smart_conn_reuse
fwconn_smart_conn_reuse = 1

I will make a change and see what happens.

Re: Packet corruption by firewall

the_rock — Mon, 04 Dec 2023 13:43:53 GMT

Definitely could be the issue. I recall spending whole night just before covid-19 hit at customer's site when they upgraded from R80.20 to R80.30 with TAC T3 and escalation and we ended up finding this setting ourselves and once turned off, it fixed the problem. It was solved later with jumbo that came out...

Cheers,

Andy

Re: Packet corruption by firewall

Timothy_Hall — Mon, 04 Dec 2023 13:48:02 GMT

Do you have SYN Defender enabled? This issue was just fixed by the most recent Jumbo HFAs:

PRJ-49379,

PRHF-30056

SecureXL

SYN Defender may not correctly handle reused connections.

Re: Packet corruption by firewall

Hugo_vd_Kooij — Wed, 06 Dec 2023 06:56:19 GMT

Will look into that today. As I followed sk24960: "Smart Connection Reuse" feature modifies some SYN packets yesterday and I still don't get dropped packets in the logs. Not even in the zdebug output.

Someone raised the default idle timmer from 1 hour to 4 hours on this firewall. And digging into the connection table I could find connections that were idle for almost 3 hours. And if I read sk65133: Connections Table Format correctly it seems these sessins have only seen thre FIN packets from the server and not from the client.

So the firewall is being stateful and keeps them open waiting for the other FIN.

May do some serious packet capturing over the course of the day to find the reason for this.

(2 hosting parties, one of which is Azure, a VPN tunnel, a MPLS link, .... only a dozen things that could go wrong here 😉 )

Re: Packet corruption by firewall

Hugo_vd_Kooij — Wed, 06 Dec 2023 07:00:06 GMT

Oh and we have another weird thing on the standby member that drops HTTPS a lot on IPS stuff like maximum header length exceeded. So instaling a Jumbo Hotfix is not on the table as an option at the moment.