Configuring dynamic Rate Limiting rules ?

[Maller]

Hi 

Is it  possible to configure  rate limiting rules in monitor mode and if a configured  rate exceeds a threshold  this rule becomes a drop rule?

thanks

[_Val_]

Could you please elaborate on that?

[Maller]

Hi Val

Last weeks we are detecting  DDos attacks against our public dns servers.

When attack is detected we apply fwaccel dos rule  to drop dns queries when rate is above 300 queries per second.

My question is  if there is a way to configure a permanent  fwaccel dos monitor rule and when rate is above a threshold this rule can change from monitor to drop mode.

Other option is to configure a drop rule without timeout but it should consume resources unnecessarily.

Thanks

[Sorin_Gogean]

Hello @Maller ,

As it seems we are facing similar HUGE DNS traffic - like ~300K connections (it varies) - I wanted to ask in regards to "Last weeks we are detecting  DDos attacks against our public dns servers." how were you detecting this ?

The way we were looking to detect, was to either use Smart Events that would be triggered when the Correlation will see HUGE traffic/connections from different public IP's , or have a BASH script that would be triggered when the number of connections goes over a pre-defined value, and collect connection table logs.

As we enabled fast_accel rules on the DNS, in order to ease the load on the GW for now, we can't use the SmartEvent, so we did use the BASH collect script . (still needing to adjust it a bit to be a sharable solution)

So if you can share on your process/way of detecting the DNS attacks, would apreciate.

As to prevent, we look into enabling pbox (penalty box) and also define connection rate-limit rules, so we can limit the number of connections, a certain machine/IP can have in a certain moment for external facing services.

Thank you,

[Maller]

Hello Sorin

We have two ways to detect it .

We have monitored  conn. table entries in gw , usually are around  90 - 100k entries and when DDos DNS attacks are active they rise more than 200k

Additionally our dns servers (infoblox) have a security module (ADP)  that detects NXdomain flooding attacks  and also incorpores a rate limiting response protection.

[Sorin_Gogean]

Understood, so the "monitored  conn. table entries in gw , usually are around  90 - 100k entries and when DDos DNS attacks are active they rise more than 200k " is smth like fw ctl conntab or similar ?

Ty,

[_Val_]

Have you looked into sk112454?

[Sorin_Gogean]

Yes @_Val_ , we already did the rate-limiting from that SK, and currently we're looking into getting penalty box implemented.

Problem we face/address, is to easily get/determine the connection rates from the logs and have them reported, in order to determine bad behavior, like this DDOS DNS that @Maller faces also.

Ty,

[_Val_]

@Sorin_Gogean could you please elaborate? 

[Sorin_Gogean]

@_Val_ , our initial problem was addressed here and after that as I needed more clarification, I asked here for more clarifications.

So, we observed on our GW's that randomly (once a day or every 2 days) we get a HUGE amount of connections - like 300K to the HIGHEST 1MIL connections . 

This was handled by the appliance, but still we were seeing impact with some legitimate traffic and increase in the memory utilization.

From there, we started to look on a way to determine the connections that were determining this behavior, and the only feasible way, was to go with a script that takes the connection list from "fw tab -u -t connections -f" and sorts it by SRC and separately by DST . So we have developed a BASH script that get's triggered if "fw ctl pstat | grep Concurrent | awk '{print $3}'" is over the 150K connections (or 200K connections)  .

Now that we have the TOP 10 from the bash script, we looked further and filtered the connections by those TOP X IP's determined above, and we've seen that in the last month, the HIGHEST connections were against our public DNS server ( seen on all continents/DataCenters) .

So right now we have a plan and a way to determine what is causing our problems.

In order to limit the impact of this RANDOM HUGE DNS traffic, we implemented some fast_accel rules (specifically for the DNS traffic), so we don't load too much the appliance for now (not sure if it's GOOD or BAD approach).

While analyzing data, we also implemented some fwaccel rules, where we capped the number of new connections per IP to 500 (not sure if it's too high or too low)  - still the fwaccel action is set to Notification right now - as we're looking into the best way to block this without impacting GOOD traffic.

So, in the end, all this was done pretty much with scripting (watching other tolls return-data), and I was hoping we could get something with the SmartEvent or Logging solution, or a way that we can count (for a predefined time) connections and report them somehow - maybe it can be included in Skyline 😊 .

Ty,

PS: sorry for the looong post...

[Maller]

Hi Sorin

Interesting , I was thinking in something similar but my scripting skills have to improve 🙂

[Sorin_Gogean]

If you did sometime in your life some programming/scripting, then most definitely you can script this in BASH.

Anyway, I will finetune my script, and share it here on the forum, as right now I'm using "fw tab -u -t connections -f" (takes a LOONG time - like 5 min) and I've seen that is easier/faster to use "fw ctl conntab"  (way faster like under 1 min).

Thank you and have a nice week,

PS: I've attached an example of the file we get by email from the affected GW/Cluster 

[Sorin_Gogean]

Hello,

Just coming back with some updates, since we faced similar issue with HIGH DNS traffic .

We've taken 2 decisions, first was to lower the UDP 53 port timer from 40 sec to 10 sec for the rule(s) that were allowing external access to our public DNS, and watching the traffic for a while, we've see that this made the "attacks" lower in current session, we went from 600K (some weeks ago) to approx. 250-300K after we applied the lower UDP53 .

Now that we manage to diminish this, we also set some fwaccel dos new-conn-rate rules with a limit of 100 new connections per second. This was kicking in when we've got new High DNS traffic, and we noticed that the timeframe of the "attack" got shorter .

Hopefully it will help you too.

Ty,

[Maller]

Hi Ty

Thanks for share your findings.. it helps me ! About  fwaccel dos rules,  did you configure them as permanent rules , right?

[Sorin_Gogean]

Hello @Maller ,

Yes, we've added them to the "$FWDIR/conf/fwaccel_dos_rate_on_install" file .

Now since we did that, we've seen CKP logs with DOS Limit at the same time we get the HIGH Peaks.

If I may, can I ask you what IP's you were seeing/or still seeing when you get your DNS peaks ? 

(I want to see if the same Top10 IP's we're seeing hitting our DNS are hitting others too)

Here you have a snapshot of one of the latest captures we've run.

Begin listing TOP X DST connections: Sun Jul 10 19:43:56 EDT 2022 on HIGHexport_2022_07_10-1943.elg

167124 12.106.164.9

Detailed listing TOP X SRC connections for 12.106.164.9 DST on HIGHexport_2022_07_10-1943.elg

2018 212.12.0.2

1793 198.142.152.21

1766 198.142.152.22

1461 59.144.144.34

1311 212.12.0.3

1219 144.202.24.50

1163 194.190.127.81

1100 194.190.127.82

1061 193.232.139.81

1033 193.232.177.51

Ty,

[Maller]

Hi

Since 2 weeks ago that dns high peaks have not ocurred again.

These ar  our top ip  sources  , there are some coincidences with yours

105.16.163.1   ge-1.ns-02-jnb.za.seacomnet.com

105.16.154.1   ge-1.ns-01-emg.za.seacomnet.com

165.227.100.51

66.109.38.242      dnscache01.firstlight.net

198.142.152.22    dns11.mel.optusnet.com.au

59.144.144.34       aes-static-034.144.144.59.airtel.in

198.142.152.21    dns10.mel.optusnet.com.au

190.21.241.2          190-21-241-2.baf.movistar.cl

194.19.2.22            slam.dns.as2116.net

195.34.108.171      195-34-108-171.network.bg

Manel

[PhoneBoy]

See if the Penalty Box meets your precise requirements: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 

[Sorin_Gogean]

@PhoneBoy , we enabled pbox on a similar situation, but for now we also enabled "monitor mode" , in order to watch what happens. Is it correct to state that pbox is only monitoring, and not dropping traffic now 🤔 ? 

so we are looking into enabling the connection rate-limiting, and penalty box for external traffic.

is the decision of rate-limit/pbox affected in any way by fast_accel rules ? 

[Expert@XxXx-FW01:0]# fwaccel dos config get     rate limit: enabled (with policy)     rule cache: enabled           pbox: enabled      deny list: enabled (with policy)     drop frags: disabled      drop opts: disabled       internal: enabled        monitor: disabled      log drops: enabled       log pbox: enabled     notif rate: 100 notifications/second      pbox rate: 500 packets/second       pbox tmo: 180 seconds [Expert@XxXx-FW01:0]#

[Expert@XxXx-FW01:0]# fwaccel dos pbox -m Penalty box monitor_only: "on" [Expert@XxXx-FW01:0]#

thank you,

[PhoneBoy]

Yeah, with monitor_only on, there are no drops taking place.
Not sure how pbox and fastaccel work together (or don't). 

[Maller]

Yes , I was reading sk112454 , maybe I  don't undesrtand correctly  but penalty box  starts to work when it detects large number of dropped packets .  But in my case , DDoS dns flooding are allowed traffic  asking our dns servers for inexistent domains ,  There  are thousand of  internet addresses  doing a "legal" query for a domain , with the difference that it does not exist.

As Sorin suggests a combination of scripting with fwaccel dos  seems to be a good option.

[Sorin_Gogean]

So, let's be on the same page/understanding.

My script is used to determine the generators of the traffic that are seen at the time it's happening, while in order to address the problem, you should use some newconnection-rate-limit and pbox. This is how we planned to...

Ty,

[Vladimir]

I really wish that CP would implement fwaccel dos and penalty box rules in UI as well as add indicators for the gateways on which those are triggered. Perhaps as a replacement for SAM Rules UI.