connectivity problems, max connections/sessions between two hosts

[Wolfgang]

We had some problems with connections between heavy communicating proxy servers. The traffic between proxyA and proxyB flows through a Maestro gateway and is inspected there. We have sometimes connectivity problems with some sessions, mostly like Videoconferencing sessions like Teams or WebEx via HTTPS. Sessions are disrupted and are working again after reconnect from the client side. Problems are mostly seen at heavy production times.

As the nature of the proxy chain we have a lot of connections / sessions only between two nodes (proxyA & proxyB). We can see on the proxy side that more then around 25.000 active sessions we have the problems. A third proxyC never reach these values and does not show the problems. The sending proxyA reports connectivity errors to proxyB in case of the problem. proxyA and proxyC are working loadbalanced and send all traffic to proxyB.

Our main question at the moment .... are there any limits for the count of connections / sessions between two hosts ? No NAT is done for this connections, straight through the gateway.

This is Maestro R81.20 with VSX (3x 9700 appliances)

[emmap]

There's not a per-host connection limit that I'm aware of, but as it's communications between two single hosts, there's only so many TCP source ports available. If the source side of the setup starts reusing source ports before fully closing out an old connection, the gateway might not like that. It should give you meaningful drop logs though.

You also might have uneven distribution issues if you don't have L4 dist enabled, as all of those connections are going to 1 SGM. 

If this is a big problem and you're not super keen on inspecting this traffic (and if the network setup supports it) then this might be a time for Maestro Fast Forward.

[Timothy_Hall]

Agree with @emmap that it is a port reuse issue due to the limited number of IP addresses involved, and that L4 distribution may help.  Check these out too:

sk184181: Intermittent client timeouts when reusing source ports through a Maestro Security Group wi...

sk24960: "Smart Connection Reuse" feature modifies some SYN packets

[Wolfgang]

Thanks @emmap and @Timothy_Hall "Smart connection reuse" was a good hint again. I remember we had these to observe in the past.

Maestro FastForward can't be a solution, because the interfaces to all proxies  are  wrp-Interfaces of different virtual switches in VSX. wrp interfaces are not supported with Maestro Fast Forward.

How about enabling L4 distribution ? We played around with that in the past but never leave this enabled because of some trouble. I understand that we can get a better traffic distribution for this type of connections but I haven't a good feeling enabling L4 distribution.

[emmap]

L4 Dist will more evenly load the connections between SGMs but it won't prevent connection reuse situations, as the reused source port would end up with the connections going to the same SGM.