RST packet dropped

[Stefano_Cappell]

Hi

I'm having issues on a specific TCP stream, which has to pass through a couple of Firewalls (one "external" and one "internal") from the Internet to a server.

Specifically, at some point (for some reason we are still investigating), the server sends a TCP RST after 200 seconds, it gets to the "internal" server (on the right in the following picture), but this packet is dropped by the "external" firewall (on the left) 

In this case, the packet I'm talking about is at 07:34:27 and is a RST ACK

Here you can see that the reason why it's getting dropped is "Server to client packet of an old TCP connection", but as you can see it's not that old, It's a few minutes old and we have already raised the tcp timeout to 8 hours in order to solve this. 

I totally can't understand why: the point is that I suspect that's the cause of the following connection attempts from the client from the same port (you can see on the left on the wireshark picture): My guess is that for some reason the "external" firewall is left totally unaware of the ending of that connection  and still try to forward the coming packets and their retransmissions on the same old source port, which are completely blocked by the "internal" firewall.

Can someone help me understand what's happening here?

thanks

[Tobias_Moritz]

Is there a chance, that you did a Access Policy install in between? If yes, sk103598 is telling you why this happens.

Short summary: If you use Rematch Connections as Connection Persistence setting, and the first packet after policy install is a server to client packet, it will be dropped. If it has a RST flag, you have problems like this. Server has closed connection, but client and external firewall do not know about. Client is trying to continue using the old connection, external firewall is allowing it, because first client to server packet after policy install brings the connection entry back but internal firewall is dropping traffic because connection entry was deleted there with the TCP-RST packet from server.

[Stefano_Cappell]

If I understand this correctly, If this is the case, what you describe should happen basically only once after an installation.

This is not a single instance issue, it's been happening since a couple of month. What I've described is just an example of something that happens more than once a day. If I understand this correctly, on a normal usage firewall, shouldn't behave like that even after an installation.

Am I wrong?

[Tobias_Moritz · Accepted solution]

Depends on how often you do Access Policy installation on your firewalls 🙂

Example:

00:00 Clients opens connection to server: SYN

00:00 External firewall allows packet and creates connection table entry.

00:00 Internal firewall allows packet and creates connection table entry.

00:00 Server answers connection attempt: SYN-ACK. Packet traverses over both firewalls back to client.

00:00 Client confirms connection: ACK. Packet traverses over both firewalls back to server. Connection is fully established on all four devices (client, external firewall, internal firewall, server).

00:xx Free packet flow over this connection.

00:10 Traffic stops (client and server do not have anything to transfer at the moment, do not use keep-alives and keep connection open).

00:15 You do an access policy install on your external firewall (it is not important what you have changed). External firewalls connection persistence is set to Rematch Connections. All current entries of the connection table are copied to a separate table and deleted from the normal connection table.

00:20 After 10 minute connection idle time, your server is closing the connection from its end (because server has configured 10 minute idle timer in this example): RST.

00:20 Internal firewall is forwarding this packet and removing its connection table entry.

00:20 External firewall is receiving this server to client packet and evaluates it agains its normal connection table: no match, because state table entry was moved to separate table. It then evaluates it against rule base: no match, because its a server to client packet and your rule is client to server (and tcp flag is not SYN). External firewall drops this packet. It never reaches the client.

00:25 Client wants to send data again over the connection, because it thinks it is still established. PSH-ACK

00:25 External firewall is receiving this client packet and evaluates it agains its normal connection table: no match, because state table entry was moved to separate table. It then evaluates it against rule base: match! It then searches the separate connection table for a former connection prior to policy install: match! It then copies the connection entry back to normal connection table, resets the connection idle timer and forwards the packet.

00:25 Internal firewall is receiving this client packet and evaluates it agains its normal connection table: no match. It then evaluates it against rule base: match! It then searches the separate connection table for a former connection prio policy install: No match, because there was no access policy install here in between and connection table entry was deleted normally at 00:20. The clients PSH-ACK packet never reaches the server. Internal firewall is silently dropping this packet. At this moment, client and external firewall are thinking the connection is still established, while internal firewall and server see this connection as closed.

00:26 Client is retransmitting its PSH-ACK packet, because it did not get an answer. Repeated.

[Stefano_Cappell]

the problem we had was way more complicated than only this aspect (too many connections for a source port pool too restricted and a smart reuse that made a mess with the combo of two firewalls with different connection tables...) but what you described was definitely happening since we correlated that every RST dropped matched an installation.

thanks