connectivity problems, max connections/sessions between two hosts

[Wolfgang]

We had some problems with connections between heavy communicating proxy servers. The traffic between proxyA and proxyB flows through a Maestro gateway and is inspected there. We have sometimes connectivity problems with some sessions, mostly like Videoconferencing sessions like Teams or WebEx via HTTPS. Sessions are disrupted and are working again after reconnect from the client side. Problems are mostly seen at heavy production times.

As the nature of the proxy chain we have a lot of connections / sessions only between two nodes (proxyA & proxyB). We can see on the proxy side that more then around 25.000 active sessions we have the problems. A third proxyC never reach these values and does not show the problems. The sending proxyA reports connectivity errors to proxyB in case of the problem. proxyA and proxyC are working loadbalanced and send all traffic to proxyB.

Our main question at the moment .... are there any limits for the count of connections / sessions between two hosts ? No NAT is done for this connections, straight through the gateway.

This is Maestro R81.20 with VSX (3x 9700 appliances)

[emmap]

There's not a per-host connection limit that I'm aware of, but as it's communications between two single hosts, there's only so many TCP source ports available. If the source side of the setup starts reusing source ports before fully closing out an old connection, the gateway might not like that. It should give you meaningful drop logs though.

You also might have uneven distribution issues if you don't have L4 dist enabled, as all of those connections are going to 1 SGM. 

If this is a big problem and you're not super keen on inspecting this traffic (and if the network setup supports it) then this might be a time for Maestro Fast Forward.

[Timothy_Hall]

Agree with @emmap that it is a port reuse issue due to the limited number of IP addresses involved, and that L4 distribution may help.  Check these out too:

sk184181: Intermittent client timeouts when reusing source ports through a Maestro Security Group wi...

sk24960: "Smart Connection Reuse" feature modifies some SYN packets

[Wolfgang]

Thanks @emmap and @Timothy_Hall "Smart connection reuse" was a good hint again. I remember we had these to observe in the past.

Maestro FastForward can't be a solution, because the interfaces to all proxies  are  wrp-Interfaces of different virtual switches in VSX. wrp interfaces are not supported with Maestro Fast Forward.

How about enabling L4 distribution ? We played around with that in the past but never leave this enabled because of some trouble. I understand that we can get a better traffic distribution for this type of connections but I haven't a good feeling enabling L4 distribution.

[emmap]

L4 Dist will more evenly load the connections between SGMs but it won't prevent connection reuse situations, as the reused source port would end up with the connections going to the same SGM. 

[Gennady]

hi!

sk184181 originated from a case I worked on with RnD (PRHF-41806).

This fix must not be used in Dual-Site environment because it overloads Sync during switchover. However, the fix resolves re-use problem by eliminating 1 HTU Sync delay when a connection is deleted from connection table on c2s SGM. The sync message then immediately sent to s2c SGM. By default, the delete notification occurs only at 1 HTU rate (delta sync interval). Insert notifications are sent immediately be default.

There should be asymmetric distribution for traffic from ProxyA to ProxyB in c2s and s2c direction, for the fix to be applicable to your situation. Otherwise, you can ignore this SK. 

If you have no NAT in your environment, then I would look into possibility to enable manual-general distribution. It decreases number of asymmetric connections which in turn would significantly decrease correction traffic and improve performance of the Security Group.

[Gennady]

Good day!

By any chance, is there a source port limitation on the Proxy servers?
You can check this way if those are UNIX based machines:

sysctl net.ipv4.ip_local_port_r

Example output:

net.ipv4.ip_local_port_range = 32768 65535

[simonemantovani]

Did you check if the issue could be related on some configuration on proxy side that defines, for example, the maximun number of connections accepted? In linux you could have this kind of behaviour due to the limit of connections configured as a default, but it's just a hypothesis

[Gennady]

Good day!

It is worth to mention that connection table size is limited by default in R81.20 VSX. Did you have a chance to check it?

Another thing is to double check Aggressive aging status in output of "fw ctl pstat" on impcated VS. "Aggressive Aging is enabled, not active" is normal output.