Many TCP First-Packet-isn't-SYN drops after upgrade to R81.10

[Jonathan]

Hi,

After upgrading from R80.20 to R81.10 we see many many TCP drops in the logs from many servers.

Drops are First Packet Isn't SYN with TCP flags are mostly FIN-ACK, ACK.

No performance or other issue with these servers.

We've never had these drops on R80.20 from these servers, and we haven't changed any topology when upgraded.

I suspect it's just cosmetic or a new logging feature issue, but not sure...

I've noticed a checkbox under Global Properties -> Stateful Inspection -> "Log on drop". I don't know if it was checked before the upgrade, maybe it explains the issue?

One thing to mention, not sure it's related - upgrade is not finished yet, we have one member upgraded and the other one is still running R80.20 but is not functioning (cpstop), so the cluster is actually broken at the moment.

Thanks

[Chris_Atkinson]

I would advise completing the upgrade and seeing if the issue persists with Jumbo T45 or later applied.

There have been situations where similar symptoms were reported and linked to:

PRJ-30820, PRHF-19417

SecureXL

In a rare scenario, after an upgrade, HTTPS traffic may be dropped.

[Jonathan]

Hi Chris,

I'm on the latest JHF (take 66). We don't have HTTPS inspection of this traffic, and also it happens on other services and ports other than 443.

I will wait however until we finish with the upgrade, probably next Sunday, and will update if it's resolved.

Thanks

[Chris_Atkinson]

Noted. There are other possible causes for these messages as the other discussion suggests. 😃

[Jonathan]

Hi,

So upgrade is all finished, but issue is the same.

Only noticable change is that 99% of the logs now have TCP Flag: RST-ACK, which I know is generally normal to see. I just don't understand how come we never saw it prior to the upgrade and how to stop seeing it in logs.

[skandshus]

seeing the same thing "all of the sudden" No solution so far.

[laurent_ragon]

HI,

Check this sk: sk137672 - How to change the 'TCP Half Closed timer' value.

with this solution we can improve the TCP close session processus.

[Dilian_Chernev]

We have upgraded R80.40 ot R81.10 10 days ago, and we are monitoring large number of First Packet Isn't SYN drops, mainly with packets with tcp flag RST-ACK.

For a monitored connections between selected hosts, the increase of dropped packets is with 15% compared with the day before upgrade. Also some application issues are reported after upgrade and seems related to the drops.

I am not sure how solution in sk137672 is supposed to control.

Any suggestions?

[Jonne_Hannon]

It appears that in my case, the connection is being expired after 15 seconds when the firewall sees the DEST-FIN.  The client (Chrome/Edge)  continues to send Keep-Alive packets for about 5 minutes, which are all dropped out of state.  The client then sends several FIN-ACK packets, which are also dropped out of state.  This is what the connection looks like before DST-FIN:

<00000000, 0a7a1550, 0000d333, ac1c2015, 00000050, 00000006; 0001c001, 40044080, 00000038, 000001cf, 00000000, 6387085d, 00000008, 13e3b1d3, d7e5c551, 00000003, ffffffff, ffffffff, ffffffff, 0000e800, 00000000, 80000000, 00000000, 00000000, ac442808, 00007f8e, 00000000, 02101801, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000; 3603/3615>

and after the DEST-FIN

<00000000, 0a7a1550, 0000d330, ac1c2015, 00000050, 00000006; 0001e001, 40044080, 00000038, 000001cf, 00000000, 6387085d, 00000000, 13e3b1d3, d7e5c551, 00000003, ffffffff, ffffffff, ffffffff, 0000e800, 00000000, 80000000, 00000000, 00000000, b8fd9088, 00007f8e, 00000000, 06ce0001, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000, 00000000; 14/15>

After the server sends the FIN-ACK to the client, the connection is updated with this 15 second timeout.  The client (Chrome/Edge) sends the first Keep-Alive packets after 45 seconds and several more Keep-Alive packets at 45 second intervals, These Keep-Alive packets are all dropped out of state, as well as subsequent FIN-ACK packets.  I'm not sure where the 15 second timeout is coming from; the TCP end session timeout is 5 seconds.  Disabling SecureXL resolves the issue.  I have a case open, but not getting anywhere quickly unfortunately.  I'm not sure if this is impacting user experience, other than causing Chrome and Edge to use more sockets than necessary and generating lots of unnecessary logs.  It seems to be that there is an issue with SecureXL expiring half-closed connections too early and would like to get to the bottom of it

[Dilian_Chernev]

Hi @Jonne_Hannon , do you have any solution about the problem ?

[Chris_Atkinson]

Please review sk180364 and see if it helps

[Alex-]

Might also be this from R81.10 Take 82.

PRJ-42445, PRHF-26215

SecureXL

The Security Gateway may prematurely expire half-closed TCP connections and drop VoIP and HTTPS packets with "First packet isn't SYN".

[Jonne_Hannon]

Hi Dilian,

sk137672 - How to change the 'TCP Half Closed timer' value is most helpful for me.  I am still investigating the hotfix from sk180364, because that hotfix seems to remove the half-closed DST-FIN connection from the SecureXL connections table, but the firewall half-closed DST-FIN connection still has the 15 second timeout resulting is many "First packet isn't SYN" drops.  I also note that sk137672 was recently updated with the following added information:

"Some of the Half closed connections ( DST-FIN ) by default will have 15 seconds timeout starting from the versions mentioned above.
To increase the default timeout to 3600 seconds for all Half closed connections please proceed with the procedure described above."

This seems to imply that the 15 second timeout is a change in behaviour.  I'm not sure what the rationale is behind expiring the half-closed DST-FIN connections so quickly.  Other than generating lots of "First packet isn't SYN" logs, I'm not sure if the user experience is impacted by expiring half-closed DST-FIN connections after 15 seconds.  Maybe a Check Point employee can explain the rationale behind this change?

Thanks,

Jonne.