First packet isn't Syn; tcp_flags: ACK

[MODBUSTCP88]

Hi All,

Our Firewall drops traffic between client and server randomly and we can't figure out why. Here are configuration and the log info found

Host A is configured as Client 
Host B is configured as Server

In the Checkpoint traffic details:
Source: Host B's IP Address
Destination: Host A's IP Address

TCP Packet out of state: First packet isn't SYN
TCP Flags: ACK

If TCP Flags is ACK, this means that the source is trying to send ACK to the destination. But the firewall blocks it because this is not following the TCP 3-way handshake. Is my understanding correct?

If true, why is the source which is configured as the server sending an ACK? Any ideas?
Thank you!

Best regards

[_Val_]

Yes, you are correct. In some cases, a recent TCP connection can be reused by an application. In this case, the client can send ACK through an existing or just currently expired session.

[MODBUSTCP88]

Thank you _Val_ for the quick reply. How can we prevent this?

In this case, the source (configured Server) was sending ACK to the destination (configured Client) which would cause the traffic drop. 

Would a probably fix be increasing the message timeout delay on the client? 

[_Val_]

This does not make any sense. Could you maybe provide some traces & logs? There is no reason for the server to send ACK to the client. It should be SYN-ACK or FIN-ACK, according to the TCP protocol rules.

[_Val_]

One more question. Is this issue by any chance related to VOIP?

[MODBUSTCP88]

Apologies the for multiple replies, not sure how that happen. I thought the first reply didnt go through.

Anyway, for your questions:

1. This is not related to VOIP. Its a communication between a control system and a field device PLC.

2. Unfortunately, we do not have any logs/traces other than the checkpoint data that I shared in the original post. 

[_Val_]

Understood. 

If this is an application behavior, there is not much you can do on the FW. You may look at this thread, option 2: https://community.checkpoint.com/t5/General-Topics/Disable-Stateful-Inspection/td-p/120107

I would start with the application though...

[MODBUSTCP88]

Not sure if this can help, attaching screenshot from checkpoint (removed sensitive info).

Thanks for sharing further information.

[_Val_]

This proves what you have said already, but does not bring any additional info, other than it seem you are using Modbus TCP over port 502.

According to the specifications, Server should not send ACK, see page 17 here: https://modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf

I suggest running traces to prove the dropped packet is indeed ACK and not ACK-FIN, when coming from the server. If it is, you need to talk to the application owners, of you can try disabling stateful inspection selectively as  mentioned above.

If server actually sends something other than a clean ACK, but FW considers it to be ACK and drops it eventually, you need to take it with TAC

[MODBUSTCP88]

Thanks _Val_, Unfortunately, we cannot disable the stateful inspection at this time. Will try Wireshark and also the vendor next. 

[_Val_]

Just to make it clear, I did not recommend you disable Stateful Inspection globally, I have provided a reference on how to do it for specific traffic on a limited scale. This is not the recommended workaround anyway, so I certainly understand your caution. 

[MODBUSTCP88]

Hi Val,

We managed to captured data with wireshark and have shared with the Vendor. 

We notice that the Server sends Keepalive packet and the firewall is blocking this. 

I read somewhere that this packet is normally an ACK. If true, is there a setting in the firewall that can allow keepalive packets through? 

The firewall team opened up all ports but it still drops the connection.

[_Val_]

This is a violation of the TCP state. Any stateful security device will drop such packets.

I do not think there is a way to disable TCP state on a specific service. 

You can try opening a TAC case to see if it is possible, but I would be very surprised with a positive outcome.

[Bob_Zimmerman]

How frequent is the keepalive? The default timeout for a connection with no traffic is one hour, so if the keepalive interval is longer than that, yes, you will see this behavior.

More generally, "First packet isn't SYN" drops for packets with only the ACK flag set are almost always one of two things

1. Really primitive firewalls (e.g, PIX) are sometimes set up to just allow any packet with ACK set with the expectation the endpoints will handle verifying TCP state. It could conceivably be an attack or scan trying to bypass such a firewall.
2. The traffic could be a legitimate reply to a connection which was valid at one time, but which has been removed from the connections table.

The second case is pretty easy to prove, and it turns the question into "Why was this connection removed from the connections table?", which is easy to answer. Connections are removed from the table in three ways:

1. When manually deleted (e.g, 'fw tab -t connections -x')
2. When the endpoints close it (either via RST or FIN)
3. After the connection's timeout (this can be the default timeout for the protocol, it can be set per-service-object, and it can be adjusted by Aggressive Aging if the firewall is under load)

So one of those three things is happening. A long-running packet capture filtered for those endpoints should tell you which one.

[MODBUSTCP88]

Hi Bob,

thank you for the detailed explanation. The keepalive is set at 30seconds at the server.  We think that the server is taking too long to respond to the client's request, so the clients closes the connection (the timeout delay at the client is set at 1000ms). This issue did not re-occur after increasing the timeout delay from 1000ms to 3000ms. 

I am trying to understand the drop details screenshot. Took one from google images below.

Does Policy Date value represents the date when Policy was implemented in the firewall that caused the drop to happen?

Thank you.

[eslopezps]

Hi there, how or where did you change the timeout delay ? i have the same problem, some packets are passed in the configured rule and other are drop by a clean up rule, and the only information we have is that "TCP packet out of state” : First packet isn´t SYN "

[Chris_Degan]

hi @_Val_  I have a where it involves VOIP with remote access VPN users who intermittently not able to make outbound call and the packets is being dropped as first packet is not syn. in coming calls work fine. no asymmatric routing. 

thank you

[MODBUSTCP88]

Thank you _Val_ for the quick reply.
In our case, it is the Configured Server (Host B) that is sending the ACK not the client. 

Would increasing timeout message delays on the client help? It is currently set at 1 second.

Thank you

[BikeMan]

Hi,

You could try sk102491.

Rgds,