topic Hanging client ports in chassis in Hyperscale Firewall (Maestro)

Hanging client ports in chassis

Kaspars_Zibarts — Fri, 14 Jun 2019 13:22:41 GMT

Here's a weekend riddle for those running scalable platforms (we're on R76 SP50 T62 with 4 SGMs) 🙂 I need a break now as it took forever to get to the truth. Will be raising case after weekend!

In nutshell, we are seeing some connections from a client to domain controller not answered (4 TCP SYNs sent and no response) so we quickly blamed MS/WinOS. But it turned out that chassis was sitting in some strange state regarding some client ports - one SGM thought that connection is idle and correction SGM still had connection in the table.

After gigabytes of packet capture we got it - this scenario was created when TCP connection is released from both client and server nearly simultaneously. So somehow connection table update fails on SGMs

As always one diagram speaks 1000 words..

thanks as always!

I know it's true because after manually deleting connection from blade 1_03 table, all works again on that port.

Re: Hanging client ports in chassis

Murad_Elmgbar — Fri, 14 Jun 2019 22:10:03 GMT

Yup, that same solution i used. once the connection is established again the issue fixed.

Re: Hanging client ports in chassis

Kaspars_Zibarts — Mon, 17 Jun 2019 10:43:14 GMT

Doesn't really work for us as it's a domain controller with millions of connections so I cannot sit there monitoring for hanging ports. Been trying to tweak the correction flows now to get the flow in a single blade but that won't fix actual underlying bug 😞