recently a customer started to complain about random traffic disruption. During investigation I have found, that reported time consists with policy installation on VSX R77.30 with HFA 338 gateway with few virtual systems. During debug I have found, that issue is related to connection rematch, which (from my point of view and understanding), does not match correctly existing connection.
;[vs_5];[tid_0];[fw4_0];fw_log_drop_ex: Packet proto=6 10.10.10.10:3389 -> 184.108.40.206:61110 dropped by fw_handle_old_conn_recovery Reason: TCP packet that belongs to an old connection;
I would expect such message related case, when policy is changed in way, that connection permitted in "old" policy is dropped in "new" policy, like sk140112 . Expected behavior should be, that during policy installation, already established connections are marked as "old". Then next packet will arrive to gateway, will be checked against new policy, when permitted marking in connection table should have status changed back to regular connection. But such has not happened in my case and I would like to know the reason. Further more, I can see this drop only in direction server->client, connection is established as client->server.
As one of solution will be to change Connection Persistence to "Keep all connections", but such I do not like due to security concerns. Secondly I do have many VSX clusters R77.30 and only two of them with HFA 338 are subject of this drop. To make it even less clear, on one gateway I can see this behavior on all virtual systems, on second cluster only on few virtual systems. Then for me it looks more like software defect and not design issue.
Do you have any tips, how to debug it?
Did I miss something?