- CheckMates
- :
- Products
- :
- Quantum
- :
- Security Gateways
- :
- Re: Configuring dynamic Rate Limiting rules ?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Configuring dynamic Rate Limiting rules ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Could you please elaborate on that?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Val
Last weeks we are detecting DDos attacks against our public dns servers.
When attack is detected we apply fwaccel dos rule to drop dns queries when rate is above 300 queries per second.
My question is if there is a way to configure a permanent fwaccel dos monitor rule and when rate is above a threshold this rule can change from monitor to drop mode.
Other option is to configure a drop rule without timeout but it should consume resources unnecessarily.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Maller ,
As it seems we are facing similar HUGE DNS traffic - like ~300K connections (it varies) - I wanted to ask in regards to "Last weeks we are detecting DDos attacks against our public dns servers." how were you detecting this ?
The way we were looking to detect, was to either use Smart Events that would be triggered when the Correlation will see HUGE traffic/connections from different public IP's , or have a BASH script that would be triggered when the number of connections goes over a pre-defined value, and collect connection table logs.
As we enabled fast_accel rules on the DNS, in order to ease the load on the GW for now, we can't use the SmartEvent, so we did use the BASH collect script . (still needing to adjust it a bit to be a sharable solution)
So if you can share on your process/way of detecting the DNS attacks, would apreciate.
As to prevent, we look into enabling pbox (penalty box) and also define connection rate-limit rules, so we can limit the number of connections, a certain machine/IP can have in a certain moment for external facing services.
Thank you,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Sorin
We have two ways to detect it .
We have monitored conn. table entries in gw , usually are around 90 - 100k entries and when DDos DNS attacks are active they rise more than 200k
Additionally our dns servers (infoblox) have a security module (ADP) that detects NXdomain flooding attacks and also incorpores a rate limiting response protection.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Understood, so the "monitored conn. table entries in gw , usually are around 90 - 100k entries and when DDos DNS attacks are active they rise more than 200k " is smth like fw ctl conntab or similar ?
Ty,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Have you looked into sk112454?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes @_Val_ , we already did the rate-limiting from that SK, and currently we're looking into getting penalty box implemented.
Problem we face/address, is to easily get/determine the connection rates from the logs and have them reported, in order to determine bad behavior, like this DDOS DNS that @Maller faces also.
Ty,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Sorin_Gogean could you please elaborate?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@_Val_ , our initial problem was addressed here and after that as I needed more clarification, I asked here for more clarifications.
So, we observed on our GW's that randomly (once a day or every 2 days) we get a HUGE amount of connections - like 300K to the HIGHEST 1MIL connections .
This was handled by the appliance, but still we were seeing impact with some legitimate traffic and increase in the memory utilization.
From there, we started to look on a way to determine the connections that were determining this behavior, and the only feasible way, was to go with a script that takes the connection list from "fw tab -u -t connections -f" and sorts it by SRC and separately by DST . So we have developed a BASH script that get's triggered if "fw ctl pstat | grep Concurrent | awk '{print $3}'" is over the 150K connections (or 200K connections) .
Now that we have the TOP 10 from the bash script, we looked further and filtered the connections by those TOP X IP's determined above, and we've seen that in the last month, the HIGHEST connections were against our public DNS server ( seen on all continents/DataCenters) .
So right now we have a plan and a way to determine what is causing our problems.
In order to limit the impact of this RANDOM HUGE DNS traffic, we implemented some fast_accel rules (specifically for the DNS traffic), so we don't load too much the appliance for now (not sure if it's GOOD or BAD approach).
While analyzing data, we also implemented some fwaccel rules, where we capped the number of new connections per IP to 500 (not sure if it's too high or too low) - still the fwaccel action is set to Notification right now - as we're looking into the best way to block this without impacting GOOD traffic.
So, in the end, all this was done pretty much with scripting (watching other tolls return-data), and I was hoping we could get something with the SmartEvent or Logging solution, or a way that we can count (for a predefined time) connections and report them somehow - maybe it can be included in Skyline 😊 .
Ty,
PS: sorry for the looong post...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Sorin
Interesting , I was thinking in something similar but my scripting skills have to improve 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you did sometime in your life some programming/scripting, then most definitely you can script this in BASH.
Anyway, I will finetune my script, and share it here on the forum, as right now I'm using "fw tab -u -t connections -f" (takes a LOONG time - like 5 min) and I've seen that is easier/faster to use "fw ctl conntab" (way faster like under 1 min).
Thank you and have a nice week,
PS: I've attached an example of the file we get by email from the affected GW/Cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Just coming back with some updates, since we faced similar issue with HIGH DNS traffic .
We've taken 2 decisions, first was to lower the UDP 53 port timer from 40 sec to 10 sec for the rule(s) that were allowing external access to our public DNS, and watching the traffic for a while, we've see that this made the "attacks" lower in current session, we went from 600K (some weeks ago) to approx. 250-300K after we applied the lower UDP53 .
Now that we manage to diminish this, we also set some fwaccel dos new-conn-rate rules with a limit of 100 new connections per second. This was kicking in when we've got new High DNS traffic, and we noticed that the timeframe of the "attack" got shorter .
Hopefully it will help you too.
Ty,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Ty
Thanks for share your findings.. it helps me ! About fwaccel dos rules, did you configure them as permanent rules , right?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Maller ,
Yes, we've added them to the "$FWDIR/conf/fwaccel_dos_rate_on_install" file .
Now since we did that, we've seen CKP logs with DOS Limit at the same time we get the HIGH Peaks.
If I may, can I ask you what IP's you were seeing/or still seeing when you get your DNS peaks ?
(I want to see if the same Top10 IP's we're seeing hitting our DNS are hitting others too)
Here you have a snapshot of one of the latest captures we've run.
Begin listing TOP X DST connections: Sun Jul 10 19:43:56 EDT 2022 on HIGHexport_2022_07_10-1943.elg
167124 12.106.164.9
Detailed listing TOP X SRC connections for 12.106.164.9 DST on HIGHexport_2022_07_10-1943.elg
2018 212.12.0.2
1793 198.142.152.21
1766 198.142.152.22
1461 59.144.144.34
1311 212.12.0.3
1219 144.202.24.50
1163 194.190.127.81
1100 194.190.127.82
1061 193.232.139.81
1033 193.232.177.51
Ty,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi
Since 2 weeks ago that dns high peaks have not ocurred again.
These ar our top ip sources , there are some coincidences with yours
105.16.163.1 ge-1.ns-02-jnb.za.seacomnet.com
105.16.154.1 ge-1.ns-01-emg.za.seacomnet.com
165.227.100.51
66.109.38.242 dnscache01.firstlight.net
198.142.152.22 dns11.mel.optusnet.com.au
59.144.144.34 aes-static-034.144.144.59.airtel.in
198.142.152.21 dns10.mel.optusnet.com.au
190.21.241.2 190-21-241-2.baf.movistar.cl
194.19.2.22 slam.dns.as2116.net
195.34.108.171 195-34-108-171.network.bg
Manel
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
See if the Penalty Box meets your precise requirements: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@PhoneBoy , we enabled pbox on a similar situation, but for now we also enabled "monitor mode" , in order to watch what happens. Is it correct to state that pbox is only monitoring, and not dropping traffic now 🤔 ?
so we are looking into enabling the connection rate-limiting, and penalty box for external traffic.
is the decision of rate-limit/pbox affected in any way by fast_accel rules ?
[Expert@XxXx-FW01:0]# fwaccel dos config get rate limit: enabled (with policy) rule cache: enabled pbox: enabled deny list: enabled (with policy) drop frags: disabled drop opts: disabled internal: enabled monitor: disabled log drops: enabled log pbox: enabled notif rate: 100 notifications/second pbox rate: 500 packets/second pbox tmo: 180 seconds [Expert@XxXx-FW01:0]# |
[Expert@XxXx-FW01:0]# fwaccel dos pbox -m Penalty box monitor_only: "on" [Expert@XxXx-FW01:0]# |
thank you,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yeah, with monitor_only on, there are no drops taking place.
Not sure how pbox and fastaccel work together (or don't).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes , I was reading sk112454 , maybe I don't undesrtand correctly but penalty box starts to work when it detects large number of dropped packets . But in my case , DDoS dns flooding are allowed traffic asking our dns servers for inexistent domains , There are thousand of internet addresses doing a "legal" query for a domain , with the difference that it does not exist.
As Sorin suggests a combination of scripting with fwaccel dos seems to be a good option.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So, let's be on the same page/understanding.
My script is used to determine the generators of the traffic that are seen at the time it's happening, while in order to address the problem, you should use some newconnection-rate-limit and pbox. This is how we planned to...
Ty,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I really wish that CP would implement fwaccel dos and penalty box rules in UI as well as add indicators for the gateways on which those are triggered. Perhaps as a replacement for SAM Rules UI.