Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Teddy_Brewski
Contributor
Jump to solution

Saturation of concurrent connections

Hello,

VSX cluster running Check Point R80.40 (Take 154) on open servers (HP) with two VSs.

On a random basis, but always during out of office hours and on weekends, we experience 1-2 minutes intermittent access due to the spikes of concurrent connections.
CPU load goes 100%, all concurrent connections are saturated and the firewall starts dropping packets. It only lasts for 2-3 minutes, and could happen once/twice per week/month. It does look like a DDoS, but it only lasts for few minutes.

I keep raising the concurrent connections value, initially from 25000 to 35000, and then to 55000, but it doesn't seem to help. We have plenty of RAM and potentially can go higher, but I'm not sure it's the right way.
I can't spot anything unusual from the SmartTracker logs during those minutes -- just regular port scans from various networks.

Any ideas/tips/hints would be greatly appreciated.

PS: Just to add that we've experienced the same with R77.30, so I don't think it's linked to the version.

PSS: The specs are: ProLiant DL360 Gen10 (Intel Xeon Gold 6144 3.50GHz (8 cores), 64GB RAM).

Thank you in advance.

0 Kudos
1 Solution

Accepted Solutions
Chris_Atkinson
Employee
Employee

What size internet link are the gateways connected to and how many users do they protect?

Do you have a number of public IPs routed towards the firewall that aren't necessarily in use?

Is there a router under your control/management located north of the firewall?

View solution in original post

0 Kudos
40 Replies
Chris_Atkinson
Employee
Employee

What size internet link are the gateways connected to and how many users do they protect?

Do you have a number of public IPs routed towards the firewall that aren't necessarily in use?

Is there a router under your control/management located north of the firewall?

0 Kudos
Teddy_Brewski
Contributor

Thank you @Chris_Atkinson 

The Internet link is 1Gb.  One the firewall side, the Internet facing part is in a bond, with 2x10G, dot1q, interfaces.

There is /20 range behind, with not everything routed towards the firewall.  The affected VS handles 8 /24 public networks.

I do have a router (Arista DCS-7020SR) under my control in front.

Thank you.

Chris_Atkinson
Employee
Employee

How are routes anchored in the environment to prevent traffic looping?

Some generic advice that might help minimize noise and prevent the traffic from getting to the Firewall in the first instance:

- Null route any of your unused public networks/subnets at the Router level

- Bogon & Martian filters / infrastructure ACLs at the Router level

 

Your ISP may partially implement something like the second point on your behalf already, some are better than others at doing so however.   

 

Refer also: sk112454 - How to configure Rate Limiting rules for DoS Mitigation (R80.20 and newer)

 

 

the_rock
Legend
Legend

Do you have this option in the cluster properties? If so, I would change it to automatic, as that way, gateway would automatically calculate cpu/memory redistribution based on amount of connections, rather than setting it up manually.

Andy

 

Screenshot_1.png

0 Kudos
Chris_Atkinson
Employee
Employee

VSX / Virtual Systems don't have this option unfortunately 

But with 64GB RAM you should feel comfortable in increasing the manual values considerably from defaults where needed.

the_rock
Legend
Legend

Ah kk, thanks Chris, good to know. Last time I worked with VSX was in R77.30, so could not recall if that option was there 🙂

0 Kudos
Teddy_Brewski
Contributor

Our local CP support proposed to keep increasing the number, so I jumped to 55000, which was still saturated,

0 Kudos
Sal_Previtera
Contributor

Check DNS traffic, lot of users may change to an external DNS or trying to hide behind encrypt DNS in order to bypass your inspection...disable any external DNS from the user traffic, if you can,

Teddy_Brewski
Contributor

Thank you for the suggestion @Sal_Previtera 

DNS traffic from internal users to external DNS servers is blocked. Also, it always happens outside of working hours (for example Friday, 11pm), with nobody present/connected in the office.

It looks like it's an external attempt (massive port scan?), but I can't get enough evidence during this several minutes incident.

0 Kudos
Chris_Atkinson
Employee
Employee

Admittedly less relevant here it seems but some also find it helpful to lower the default timeout for the DNS service.

0 Kudos
the_rock
Legend
Legend

Not to sound ironic now, but that would be same if car mechanic told you to keep adding oil constantly, though you know its leaking, that wont fix the problem permanently, its not even a good workaround. We need to find out WHY its happening, so it can be fixed once for all.

Any clue when this started? Were there any changes made that you can recall that would have caused such a behavior?

0 Kudos
Sal_Previtera
Contributor

My response was based on my personal experience with traffic originating internally for DNS traffic, with saturated connections on our Free internet supplied to our customers...may I should have it stated that earlier. 

Most user were trying to bypass our inspection with external or encrypt DNS, now that some browsers use.

In your case, if it is external traffic reaching your firewalls, may need to use some anti-DDOS or suspicious rules...

You may want to log external traffic being dropped until you find the source or multiple sources...if it is not logged. 

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_LoggingAndMonitoring_AdminGuide/To...

 

the_rock
Legend
Legend

Hey Sal,

No no, your response was excellent, I commented on the suggestion Teddy gave from local CP office to keep increasing the amount of connections limit. Personally, I dont think thats even good workaround. What you gave makes total sense.

0 Kudos
Teddy_Brewski
Contributor

I fully agree. 😀

I've seen it with R77.30 before (a year ago).   Different hardware though (also slighty oversized when it comes to RAM) and no VSX.  It was the same behavior, and if I'm not wrong, it was fixed (or hid?) by choosing Automatic Capacity Calculation, so it's quite possible it was always there.  What changed now is the frequency.

 

(1)
the_rock
Legend
Legend

Can you send us output of below please Teddy?

fw ctl pstat

fw ctl multik print_heavy_conn

Andy

Teddy_Brewski
Contributor

@the_rock wrote:

Can you send us output of below please Teddy?

fw ctl pstat

fw ctl multik print_heavy_conn

Andy


Thank you Andy.  Here it is:

# fw ctl pstat

Virtual System Capacity Summary:
  Physical memory used:   4% (2666 MB out of 53949 MB) - below watermark
  Kernel   memory used:   1% (708 MB out of 53949 MB) - below watermark
  Virtual  memory used:   2% (1405 MB out of 62200 MB) - below watermark
     Used: 237 MB by FW, 1152 MB by zeco
  Concurrent Connections: 28% (15713 out of 54900) - below watermark
  Aggressive Aging is enabled, not active

Kernel memory (kmem) statistics:
  Total memory  bytes  used: 102148474   peak: 145904923
  Allocations: 0 alloc, 0 failed alloc
               0 free, 0 failed free

Cookies:
        2273050826 total, 0 alloc, 0 free,
        841513 dup, 4160716724 get, 8779887 put,
        360874770 len, 1787451782 cached len, 0 chain alloc,
        0 chain free

Connections:
        168007326 total, 107162621 TCP, 47762740 UDP, 13044116 ICMP,
        37849 other, 1769045 anticipated, 29370 recovered, 15713 concurrent,
        54900 peak concurrent

Fragments:
        108 fragments, 24 packets, 0 expired, 0 short,
        0 large, 0 duplicates, 0 failures

NAT:
        2645093/0 forw, 2036238/0 bckw, 2773631 tcpudp,
        2115370 icmp, 1388763-712045 alloc

Sync: Run "cphaprob syncstat" for cluster sync statistics.

 

'fw ctl multik print_heavy_conn' returned nothing.

~15k connections is how our normal evening looks like.

0 Kudos
the_rock
Legend
Legend

Ok, that looks good, I mean, 28% is nothing, wayyy below limit. By the way, can you also run cpview and tab between different fields to see if there is anything of interest there?

Andy

0 Kudos
Teddy_Brewski
Contributor
 

This is how CPU and concurrent connections look like at the time of the incident.  I extracted it from 'cpview -t'.

2-3 minutes before and after were perfectly normal values.

CaptureCP.PNG

the_rock
Legend
Legend

Yea, that looks fine to me as well. Maybe worth TAC investigating, but then again, if issue is not there, except weekend, not sure how much they can do either. Is it possible to have someone available when this occurs on the weekend or thats sort of random too, does not always happen at the same time?

Andy

0 Kudos
the_rock
Legend
Legend

Hey @Teddy_Brewski , something to maybe verify. I found an email from few years ago where I worked with customer who had similar issue (though this was Fortinet) and they discovered it was a scanning machine in their network causing this, since it was scheduled to run scans of large portion of their network on the weekends.

Not saying thats the case with you, but wanted to confirm.

0 Kudos
Teddy_Brewski
Contributor

I thought about that too, including some backups happening in the network, but unfortunately, can't spot the pattern yet: it's not happening on a recurring basis, always during weekends and in the evenings, but random dates. Too quick to catch in real time -- by the time I login it's already back to normal.

0 Kudos
the_rock
Legend
Legend

Hm, I wonder if there is a way to set up some sort of monitoring on it to catch more info that would be useful.

0 Kudos
Timothy_Hall
Champion
Champion

Short answer: You need to download and fire up @HeikoAnkenbrand's awesome econn tool (Easy Tool - R81.20 Real time connection table analysis v5.0 ) while the overflow is happening, it will slice and dice the connections table multiple ways and allow you to identify the "top talkers" or whatever traffic is causing the connections table overflow.  Never tried it on VSX but as long as you are in the proper VS I assume it would work.

Long Answer: This exact scenario covered extensively in my Gateway Performance Optimization R81.20 class and actually experienced/corrected in a lab exercise as well.  Here are the relevant pages:

conn1.pngconn2.pngconn3.pngconn4.png

 

Updated 2023 IPS/AV/ABOT R81.20 Course now
available at maxpowerfirewalls.com
(1)
Sorin_Gogean
Advisor

Hello @Teddy_Brewski ,

 

As we were facing some HIGH Connection Spikes in our environment, and we were blind - as in not be able to see the traffic at that particular moment it was happening - we've created a script that runs in the background and watches the current connections through time.

Collecting those current connections values, we calculate an average for the last hour, and in case the number of connections from this moment is bigger than the 1hour average plus 25% (or whatever value you can consider a valid increase) then we trigger the data gathering.

Data gathering, it means it's dumping all "$FW ctl conntab " into a file, then we parse that file and report on the top 5 IP's that are source and top 5 IPs that are destination and more than that, on each one from those 5 we report top 10 IP's that hare high connetions.

In other words, if you have a public DNS behind the CheckPoint, and that has high number of connections, then we will show top 10 source IP's towards that DNS server.

 

This script helped us see what were the spikes we encounter - like 1 - 1.5MIL connections that we got lowered to 200-300K connections now 😁

 

All those things are done in the background, and the report get's emailed, plus you can email the FWL connection export (still it's an HUGE file - like 40 - 50Mb) .

 

So, if this it would help, then let me know and we can discuss this week, share the script here and walk over .

 

Thank you,

PS: we intend to share that script here on CheckMates, but there are still some parts in work ....

Teddy_Brewski
Contributor

Thank you @Sorin_Gogean -- greatly appreciated!  I'd be happy to try your script -- we're on R80.40.  If it can be only shared privately for the time being, I can contact you offline.

0 Kudos
Sorin_Gogean
Advisor

@Teddy_Brewski , I'll share the script with you, no worries.
Let me just adapt it to your needs and then we can meet for some time and guide you through - it's quite easy.

Like I said, we did it for our needs and I would really want to check it on other environments .

 

Ty,

(1)
_Val_
Admin
Admin

Stressing the point of sharing the script in the community, @Sorin_Gogean 🙂

0 Kudos
Sorin_Gogean
Advisor

@_Val_ , we'll do that for sure, if you have some guidelines that I cold look on how the code should be, it would be great.

0 Kudos
the_rock
Legend
Legend

Happy to try it in my lab if you are willing to share 🙂

0 Kudos