topic Re: Saturation of concurrent connections in Firewall and Security Management

Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 09:19:06 GMT

Hello,

VSX cluster running Check Point R80.40 (Take 154) on open servers (HP) with two VSs.

On a random basis, but always during out of office hours and on weekends, we experience 1-2 minutes intermittent access due to the spikes of concurrent connections.
CPU load goes 100%, all concurrent connections are saturated and the firewall starts dropping packets. It only lasts for 2-3 minutes, and could happen once/twice per week/month. It does look like a DDoS, but it only lasts for few minutes.

I keep raising the concurrent connections value, initially from 25000 to 35000, and then to 55000, but it doesn't seem to help. We have plenty of RAM and potentially can go higher, but I'm not sure it's the right way.
I can't spot anything unusual from the SmartTracker logs during those minutes -- just regular port scans from various networks.

Any ideas/tips/hints would be greatly appreciated.

PS: Just to add that we've experienced the same with R77.30, so I don't think it's linked to the version.

PSS: The specs are: ProLiant DL360 Gen10 (Intel Xeon Gold 6144 3.50GHz (8 cores), 64GB RAM).

Thank you in advance.

Re: Saturation of concurrent connections

Chris_Atkinson — Mon, 20 Mar 2023 12:08:25 GMT

What size internet link are the gateways connected to and how many users do they protect?

Do you have a number of public IPs routed towards the firewall that aren't necessarily in use?

Is there a router under your control/management located north of the firewall?

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 13:27:12 GMT

Do you have this option in the cluster properties? If so, I would change it to automatic, as that way, gateway would automatically calculate cpu/memory redistribution based on amount of connections, rather than setting it up manually.

Andy

Re: Saturation of concurrent connections

Chris_Atkinson — Mon, 20 Mar 2023 13:34:33 GMT

VSX / Virtual Systems don't have this option unfortunately

But with 64GB RAM you should feel comfortable in increasing the manual values considerably from defaults where needed.

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 13:39:29 GMT

Ah kk, thanks Chris, good to know. Last time I worked with VSX was in R77.30, so could not recall if that option was there 🙂

Re: Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 15:29:59 GMT

Thank you @Chris_Atkinson

The Internet link is 1Gb. One the firewall side, the Internet facing part is in a bond, with 2x10G, dot1q, interfaces.

There is /20 range behind, with not everything routed towards the firewall. The affected VS handles 8 /24 public networks.

I do have a router (Arista DCS-7020SR) under my control in front.

Thank you.

Re: Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 15:33:29 GMT

Our local CP support proposed to keep increasing the number, so I jumped to 55000, which was still saturated,

Re: Saturation of concurrent connections

Sal_Previtera — Mon, 20 Mar 2023 15:56:38 GMT

Check DNS traffic, lot of users may change to an external DNS or trying to hide behind encrypt DNS in order to bypass your inspection...disable any external DNS from the user traffic, if you can,

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 19:04:02 GMT

Not to sound ironic now, but that would be same if car mechanic told you to keep adding oil constantly, though you know its leaking, that wont fix the problem permanently, its not even a good workaround. We need to find out WHY its happening, so it can be fixed once for all.

Any clue when this started? Were there any changes made that you can recall that would have caused such a behavior?

Re: Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 17:00:46 GMT

Thank you for the suggestion @Sal_Previtera

DNS traffic from internal users to external DNS servers is blocked. Also, it always happens outside of working hours (for example Friday, 11pm), with nobody present/connected in the office.

It looks like it's an external attempt (massive port scan?), but I can't get enough evidence during this several minutes incident.

Re: Saturation of concurrent connections

Sal_Previtera — Mon, 20 Mar 2023 17:54:20 GMT

My response was based on my personal experience with traffic originating internally for DNS traffic, with saturated connections on our Free internet supplied to our customers...may I should have it stated that earlier.

Most user were trying to bypass our inspection with external or encrypt DNS, now that some browsers use.

In your case, if it is external traffic reaching your firewalls, may need to use some anti-DDOS or suspicious rules...

You may want to log external traffic being dropped until you find the source or multiple sources...if it is not logged.

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_LoggingAndMonitoring_AdminGuide/Topics-LMG/Monitoring-Suspicious-Activity-Rules.htm

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 17:59:59 GMT

Hey Sal,

No no, your response was excellent, I commented on the suggestion Teddy gave from local CP office to keep increasing the amount of connections limit. Personally, I dont think thats even good workaround. What you gave makes total sense.

Re: Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 18:17:55 GMT

I fully agree. 😀

I've seen it with R77.30 before (a year ago). Different hardware though (also slighty oversized when it comes to RAM) and no VSX. It was the same behavior, and if I'm not wrong, it was fixed (or hid?) by choosing Automatic Capacity Calculation, so it's quite possible it was always there. What changed now is the frequency.

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 18:29:17 GMT

Can you send us output of below please Teddy?

fw ctl pstat

fw ctl multik print_heavy_conn

Andy

Re: Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 18:42:30 GMT

@the_rock wrote:
Can you send us output of below please Teddy?
fw ctl pstat
fw ctl multik print_heavy_conn
Andy

Thank you Andy. Here it is:

# fw ctl pstat Virtual System Capacity Summary: Physical memory used: 4% (2666 MB out of 53949 MB) - below watermark Kernel memory used: 1% (708 MB out of 53949 MB) - below watermark Virtual memory used: 2% (1405 MB out of 62200 MB) - below watermark Used: 237 MB by FW, 1152 MB by zeco Concurrent Connections: 28% (15713 out of 54900) - below watermark Aggressive Aging is enabled, not active Kernel memory (kmem) statistics: Total memory bytes used: 102148474 peak: 145904923 Allocations: 0 alloc, 0 failed alloc 0 free, 0 failed free Cookies: 2273050826 total, 0 alloc, 0 free, 841513 dup, 4160716724 get, 8779887 put, 360874770 len, 1787451782 cached len, 0 chain alloc, 0 chain free Connections: 168007326 total, 107162621 TCP, 47762740 UDP, 13044116 ICMP, 37849 other, 1769045 anticipated, 29370 recovered, 15713 concurrent, 54900 peak concurrent Fragments: 108 fragments, 24 packets, 0 expired, 0 short, 0 large, 0 duplicates, 0 failures NAT: 2645093/0 forw, 2036238/0 bckw, 2773631 tcpudp, 2115370 icmp, 1388763-712045 alloc Sync: Run "cphaprob syncstat" for cluster sync statistics.

'fw ctl multik print_heavy_conn' returned nothing.

~15k connections is how our normal evening looks like.

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 18:51:23 GMT

Ok, that looks good, I mean, 28% is nothing, wayyy below limit. By the way, can you also run cpview and tab between different fields to see if there is anything of interest there?

Andy

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 19:06:07 GMT

Hey @Teddy_Brewski , something to maybe verify. I found an email from few years ago where I worked with customer who had similar issue (though this was Fortinet) and they discovered it was a scanning machine in their network causing this, since it was scheduled to run scans of large portion of their network on the weekends.

Not saying thats the case with you, but wanted to confirm.

Re: Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 19:31:18 GMT

This is how CPU and concurrent connections look like at the time of the incident. I extracted it from 'cpview -t'.

2-3 minutes before and after were perfectly normal values.

Re: Saturation of concurrent connections

the_rock — Mon, 20 Mar 2023 19:43:44 GMT

Yea, that looks fine to me as well. Maybe worth TAC investigating, but then again, if issue is not there, except weekend, not sure how much they can do either. Is it possible to have someone available when this occurs on the weekend or thats sort of random too, does not always happen at the same time?

Andy

Re: Saturation of concurrent connections

Teddy_Brewski — Mon, 20 Mar 2023 19:50:25 GMT

I thought about that too, including some backups happening in the network, but unfortunately, can't spot the pattern yet: it's not happening on a recurring basis, always during weekends and in the evenings, but random dates. Too quick to catch in real time -- by the time I login it's already back to normal.