Connection rematch after policy installation

[Martin_Oles]

Hi,

recently a customer started to complain about random traffic disruption. During investigation I have found, that reported time consists with policy installation on VSX R77.30 with HFA 338 gateway with few virtual systems. During debug I have found, that issue is related to connection rematch, which (from my point of view and understanding), does not match correctly existing connection.

;[vs_5];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.10.10.10:3389 -> 20.20.20.20:61110 dropped by fw_handle_old_conn_recovery Reason: TCP packet that belongs to an old connection;

I would expect such message related case, when policy is changed in way, that connection permitted in "old" policy is dropped in "new" policy, like sk140112 . Expected behavior should be, that during policy installation, already established connections are marked as "old". Then next packet will arrive to gateway, will be checked against new policy, when permitted marking in connection table should have status changed back to regular connection. But such has not happened in my case and I would like to know the reason. Further more, I can see this drop only in direction server->client, connection is established as client->server.

As one of solution will be to change Connection Persistence to "Keep all connections", but such I do not like due to security concerns. Secondly I do have many VSX clusters R77.30 and only two of them with HFA 338 are subject of this drop. To make it even less clear, on one gateway I can see this behavior on all virtual systems, on second cluster only on few virtual systems. Then for me it looks more like software defect and not design issue.

Do you have any tips, how to debug it?

Did I miss something?

Thank you.

[G_W_Albrecht]

Maybe this can help: sk103598: Connectivity Issues after Policy Install

[Martin_Oles]

I do have different scenario, already checked this possibility. What I am observing is, that connection server->client is not correctly re-matched, even still permitted in rulebase.

Thank you.

[Václav_Brožík]

sk103598: Connectivity Issues after Policy Install states in the Scenario 2:

Note: Check Point Security Gateway never re-matches Server-to-Client connections (unless it is VPN RDP packet):

• If a Server-to-Client packet in question is a TCP, then Security Gateway mangles the packet.
• If a Server-to-Client packet in question is a UDP, then Security Gateway simply drops it.

I can confirm this behaviour. UDP packets from server to client are dropped after policy installation until a client to server packet is seen. In my experience this causes RDP over UDP to freeze and start a new session automatically. When using RDP through CyberArk then we are unlucky because CyberArk refuses to continue with the new session.

I tried to use:

fw ctl set int fwconn_rematch_udp_s2c_old_conn 1
fw ctl set int fwconn_rematch_udp_s2c_old_conn_by_service 3389

but it did not change the behaviour.

[Vladimir]

You probably have done it already, but if not, please verify clock sync on the problematic units.

Ideally, those should be using NTP.

This next suggestion does nothing to help you troubleshoot, but if you are talking about drops on particular protocols or services, you can configure connection persistence on per-service bases, no need to go global on it, unless warranted. 

[Martin_Oles]

I have checked NTP, correct on all affected clusters, members are correctly syncing time from the same source.

I can see drop across all protocols, all of them TCP, depends on customer, for example SSH, HTTP, HTTPS and some special ones. So far I did not find no relation to protocol, only direction, as I mentioned above, always server->client traffic. Tricky is, that there is 2k connections, in theory I might try to run tcpdump with "fw ctl zdebug -v x + drop", but I have find it very hard. Now I am working on connection, which is always there, for example between two particular servers. But I am afraid, that it just confirm current observation.

Thank you.