topic Re: Concurrent connections drastically increase after switch replacement in Firewall and Security Management

Concurrent connections drastically increase after switch replacement

Quentin_Antrim — Fri, 18 Dec 2020 17:54:51 GMT

I have a cluster of two CheckPoint 13000 appliances running R80.30.

Originally, on the internal side, they were connected to a core cluster of Cisco 6509 switches, each firewall connected to one of the two 6509 switches.

Just recently the core 6509 switches were replaced with a core cluster of Cisco Nexus 9500 switches, each firewall connected to one of the Nexus 9500 switches.

At the time of the replacement of 6509 switches with the 9500 switches, our average and peak connections almost doubled.

Whereas our previous normal peak would be 60K connections, our new peak became 100K connections, causing us to increase our concurrent connections max limit because of this unexpected increase.

Looking for any help in possible cause of this issue. Has anybody seen anything similar before, and what was the cause/fix?

Also trying to figure out how I can really tell what that increase in connections would be. What could I look for/at to determine what those roughly extra 40K of connections are in the firewall?

Thanks.

Quentin

Re: Concurrent connections drastically increase after switch replacement

G_W_Albrecht — Fri, 18 Dec 2020 18:07:39 GMT

My only suggestion is to check differences in the switch configs. But i fear that had been done already.

Re: Concurrent connections drastically increase after switch replacement

Quentin_Antrim — Fri, 18 Dec 2020 18:34:26 GMT

Correct. Thanks.

Re: Concurrent connections drastically increase after switch replacement

FraP — Fri, 18 Dec 2020 19:26:11 GMT

I seen a similar behaviour on a couple of nexus 9k and the issue was on switch side where for some reason packets was duplicated.
I don't know your topology so i can assume nothing, but:
Did you check if the arp table is compliant with your expectetion either on switches and firewall side?
Maybe a packet capture can help to identify duplicate packets and "netstat -ni" in expert mode to figureout if you can see error or drop on the firewall interfaces

Re: Concurrent connections drastically increase after switch replacement

Quentin_Antrim — Fri, 18 Dec 2020 20:18:06 GMT

Thanks. Yes, due to the magnitude of the increase I'm thinkng a duplicate packet issue also. I've already done some packet captures but haven't been able to determine anything yet. No errors or drop in netstat -ni. Nothing standing out in ARP table on FW, yet. Thanks for the suggestions.