connectivity problem with two proxies and a gateway in between

[Wolfgang]

Dear CheckMates,

we had a strange problem with connectivity between two proxy.

Connection looks like this:
clients => proxyA => routerA => VSX-gateway => routerB => proxyB => Internet

The clients connect to their proxyA, proxyA forwards all requests to proxyB and then proxyB sends all to the whole world.
VSX-gateway is in the middle.
Now we have massive connectivity problems between proxyA and proxyB. Website loading is very slow or can't be shown.
Sometimes proxyA reports connectivity loss to proxyB. Problems occurs only during production hours. If only 100users
are online everything is fine. With 1200users the connectivity problem occurs.

- no drop logs on the VSX-gateway for this connection, not in the logs not with fw ctl zdebug drop
- only conenction proxyA <=> proxyB involved, anything else works fine
- CPU utilization around 40% on all 16 cores, no 100% spikes
- all interfaces are fine, no rx-drops, LACP is ok
- disabling URLF, APPCL and all TP-Features (IPS, ABot etc.) doesn't help
- sim fast_accel rule for this connection doesn't help

What was changed ?
We replaced proxyA with a new vendor, now it's squid, former vendor Microsoft TMG.
And additional much more connections now, because of the homeworkers.

Seems the new proxy is the problem but we tested, analyzed and can't find anything wrong.
We had the chance to test the connection proxyA <=> proxyB via a second routing way without VSX-gateway.
Yeah, this works great, but is not what we want.

It seems a problem is with the high number of connections between the two proxies but I'm not aware of any limitations for such a case.
If we reset all connections for these proxies on the VSX-gateway, it works for 2-4minutes and then problem occurs again.

We found Latency and/or packet loss for traffic which passes through a Virtual Switch in a VSX Gateway , which described exactly our environment but we are on the latest R80.10 Jumbo.

ProxyB is working well, there are other proxies forwarding their traffic to proxyB without routing via VSX-gateway.

TAC is involved, but maybee someone has an idea what's going wrong?

thanks
Wolfgang

[Vladimir]

Doozy...

Check if Squid has "dns_v4_first on" in the /etc/squid/squid.conf and that the "X-forwarded for" is configured identical to M-TMG.

...and, of course, that its DNS resolution in general is working fast.

What are you running Squid on?

[Wolfgang]

Thanks @Vladimir . dns_v4_first is on.

But because proxyA forwards all request to proxyB only proxyB does DNS-resolving for all the websites.

Wolfgang

[Wolfgang]

Hello CheckMates,

we found a solution for our problem but I don't understand what happens.

We decreased the "Virtual session timeout" from default 3600s to 300s for the specific service TCP/8080. Everything was fine and running smooth.

Another finding..... We did an upgrade of the gateway  from to R80.10 to R80.40. With the upgrade the global parameter "TCP end timeout" was changed to 5s for gateways higher R80.20. With this setting we got back again the massive performance degradation. Setting these back to the customers value from R80.10, which was 50s, everything was fine.

I'm a little bit confused, how changing these values affects the connectivity between both proxy hosts.

Anyone played around with changes for "tcp start timeout" and "tcp end timout" values ?

Wolfgang