SecureXL DoS Rate Limiting (samp rules)

[Egenity]

I have been working a lot with the rate limiting rules via the "fw samp" CLI interface, but unfortunately I cannot get the gateway to actually enforce them.  It appears SecureXL is very unhappy when I try to enable rate limiting:

[Expert@PROD-FW02a:0]# fwaccel dos config set --enable-rate-limit
ERROR: No rate limiting policy is installed, can't enable.

What exactly is the "rate limiting policy" it is referring to?  

I have dug fairly deep in documentation, sks, etc. and cannot figure out what triggers the rate limiting capabilities of SecureXL to turn on, based on policy settings.  I also thought maybe enabling QoS blade and the QoS policy component would trigger things, but it had no effect on things.

Of course, this same status is reflected when you query the configuration (fwaccel dos config get):

rate limit: disabled (without policy)
pbox: disabled
blacklists: disabled
drop frags: disabled
drop opts: disabledfwacc
internal: disabled
monitor: disabled
log drops: enabled
log pbox: enabled
notif rate: 100 notifications/second
pbox rate: 500 packets/second
pbox tmo: 180 seconds

The gateways are R80.30 5800 appliances.

[Timothy_Hall]

Hi Adam,

You need to add a rate-limiting rule first before you can enable enforcement.  Please see this article:

https://community.checkpoint.com/t5/General-Management-Topics/How-to-completely-exclude-some-specifi...

The commands have changed slightly in R80.20+; substitute "fw samp add quota" with "fw sam_policy add quota" and "cat /proc/ppk/dos" with "fwaccel dos stats get".  You can also use "fw sam_policy get" to verify a quota rule in R80.20+ after you have added it.

[Egenity]

Sorry, I didn't make that totally clear in my initial post. I have a ton of samp rules configured, and verified by fw samp get output. However, it still refuses to kick in.

[Timothy_Hall]

By default samp rules will only apply to traffic traversing interfaces defined as External in the firewall topology, have you done a fwaccel dos config set –-enable-internal yet?

[Egenity]

Well, in this case, it is really only intended for external interface, so that angle shouldn't matter at this point.

BACKGROUND: Customer wanted to implement an IP block list from a feed. I modified some scripts that Check Point originally supplied in an SK to work properly with the feed and create the SAM rules. That part works perfectly. The gateway just won't actually do anything with them.

Small example of configured SAM policy rules:



[Expert@MDC-PROD-FW02a:0]# fw samp get | more

operation=add uid=<5e09eea8,000020a9,0501010a,0000089f> target=all timeout=1367 action=drop log=log comment=intelligo_ip_block service=any source=range:

1.4.244.35-1.4.244.35 pkt-rate=0 req_type=quota

operation=add uid=<5e09eea8,000020ab,0501010a,0000089f> target=all timeout=1367 action=drop log=log comment=intelligo_ip_block service=any source=range:

1.4.246.250-1.4.246.250 pkt-rate=0 req_type=quota

operation=add uid=<5e09eea8,000020ac,0501010a,0000089f> target=all timeout=1367 action=drop log=log comment=intelligo_ip_block service=any source=range:

[Egenity]

After a lot of troubleshooting, it appears to be a size limitation.  When I remove the long list and simply configure a test rule, the rate-limiting fires up.

The /var/log/messages gave some clues:

Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_db_rate_rset_rules_alloc (dos_db.c:2838): halloc failed: size=397592
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_db_rate_policy_alloc (dos_db.c:3518): dos_db_rate_rset_alloc failed
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_db_rate_install (dos_db.c:4049): dos_db_rate_policy_alloc failed
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_q_rate_install (dos_q.c:1257): dos_db_rate_install
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [fw4_0];cphwd_api_q_request_blocking: SecureXL device responded with an error (CPHWD_API_RESPONSE_ERROR). Retry = 0
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [fw4_0];ERROR: vs0: i0: cphwd_dos_ioctl_rate_install_g (cphwd_dos_ioctl.c:422): cphwd_dos_q_request_blocking: sxl_dev_id=0

The block list the customer wants implemented currently has 49,000+ entries (IP ranges).

TAC is a bit perplexed (6-0001867439), and I am not sure they totally understand the issue.  

I am curious if there are some adjustments somewhere to accommodate this.

[Timothy_Hall]

Looking at your sample rules, it appears you are trying to perform a block with quotas by setting a packet rate of 0 for a single IP address in each statement.  The number of quota rules you are trying to install appears to be exceeding some kind of fixed memory/table size in SecureXL, and I don't see any SecureXL kernel variables exposed that could be tweaked to increase the limit.

If you are doing a packet rate of zero to implement a block for all your samp rules, could you perhaps use the new fwaccel dos blacklist command added in R80.20 instead?  It is a much simpler feature and may have higher limits for the number of entries you can add.

[HeikoAnkenbrand · Accepted solution]

Hi @Egenity 

to fwaccel dos blacklist read more here: R80.x - Performance Tuning Tip - DDoS „fw sam“ vs. „fwaccel dos“ 

The SecureXL penalty box is a mechanism that performs an early drop of packets arriving from suspected sources. This mechanism is supported starting in R75.40VS.

Why not sam policy rules?

The SAM policy rules consume some CPU resources on Security Gateway. We recommend to set an expiration that gives you time to investigate, but does not affect performance. The best practice is to keep only the SAM policy rules that you need. If you confirm that an activity is risky, edit the Security Policy, educate users, or otherwise handle the risk. Or better use SecureXL penalty box from a performance point of view.

The purpose of this feature is to allow the Security Gateway to cope better under high load, possibly caused by a DoS/DDoS attack. These commands „fwaccel dos“ and „fwaccel6 dos“  control the Rate Limiting for DoS mitigation techniques in SecureXL on the local security gateway or cluster member.

In version R80.20, the penalty box feature is now supported in VSX mode and each virtual system can be independently configured for penalty box operation.

Attention!

In R80.20, all "sim erdos" commands are no longer supported. They have been replaced with equivalent commands which can be found under "fwaccel dos". Penalty box is configured separately for IPv4 and IPv6. IPv4 configuration is performed using the "fwaccel dos" command. IPv6 configuration is performed using the "fwaccel6 dos" command.

[Egenity]

I will examine the blacklist function, but I suspect the same type of limitation may be present.  I went with the zero rate limit  process, as described in "sk103154 - How to block traffic coming from known malicious IP addresses" -- it appears to be what Check Point recommends.

The other method I tested was a script that builds a dynamic object with all the IP ranges from the black list.  It was based on the script and idea from openDBL (discussed in another thread here).  Then just referencing the object in the rulebase from drop/log action.

Although, the concept worked, the dyanmic_objects process was a little clunky and performed poorly with 49000+ entries which is why I started down the SecureXL DoS layer implementation.  Also, I could never get the dynamic_objects API to pull in all the entries, it would always reject about 5% of the ranges.

[HeikoAnkenbrand]

From a performance point of view, I would always use fwaccel dos.

[Eric_Dale]

Hi,

A major update to sk112454 was just published.   It provides a lot more detail regarding DOS/Rate limiting rules, blacklist, and penalty box.  Hopefully it will help.

If you are just trying to block specific source IP addresses, I recommend using fwaccel dos blacklist.  Per the sk, there is a hotfix available that will scale the blacklist to millions of IPs.   It will be rolled into the R80.20 R80.30 JHF soon.

I can confirm that fw samp rules tend to have resource allocation issues when using more than about 10,000 rules.  The root cause is memory allocation failures in the kernel.   The work-around is to use the blacklist (create fewer fw samp rules).

[Timothy_Hall]

Wow, the updated SK answered all my questions and then some.  Great job!

[Egenity]

Has the hotfix been rolled in to R80.30 JHF 155?  I looked down the list of fixes, but nothing stood out as matching.

Adam

[Eric_Dale]

Not delivered yet.  It is approved and I expect it will be included in the next ongoing take.

[Egenity]

Just to confirm, is the fix in R80.40 release already?

[Eric_Dale]

Yes - it is in R80.40 already.   Also an easier to use command line for managing fw samp rules.  See  "fwaccel dos rate --help"

[Luis_Miguel_Mig]

Securexl ddos is a great feature and it works really well. I love it. 
However it is a pity that the logs generated by securexl DDOS are not indexed and therefore we have not visibility over the feature.
The reality is that it is quite risky to put in production a feature like this without logging/visibility.
When will the logs be indexed? It is very important for  a lot of customers I think.

[Eric_Dale]

I agree - the logs need to be easily searchable.   I know that there is some minimal indexing, but perhaps it is not enough.

Log indexing is outside my area of expertise - if you could provide some details on the limitations you are encountering, and suggestions on what improvements are needed, I will see what can be done.

[Luis_Miguel_Mig]

Well I think it is mandatory to be able to to search by source and destination ip for troubleshooting purposes.

But in terms of monitoring we need to be able to identify this type of alerts. The best and easiest way I can think is with the comment and name that fwaccel dos allows you to set   with -c and -n. 
This way we could totally control the number of fwaccel dos, we could create graphs to track it, etc.

[Eric_Dale]

Making logs searchable by comment/name is needed and is already in our list of things to do. I don't have an specific timeframe, but your request should help move it higher in the priority queue.

I confirmed that the logs are already searchable by source/dest IP in SmartConsole. I tested with R81.10 and also R80.20 and it seems to be working.

[Luis_Miguel_Mig]

Thanks Eric.
I guess that you mean that general firewall logs are searchable by IP.
However logs coming from securexl dos are not searchable by IP - tested in R80.40.
I guess that the only way to search them is by alert as your colleague Chris Martel suggested.

[Luis_Miguel_Mig]

I have just tested in R81 and yeah you can filter by ip.
I have configured "-a l" and you can filter by alert

The problem is that the comment field at smartconsole logs shows  the uid of the securexl dos rate limit rule instead of the comment or the name configured

It seems that the comment field is indexed. I can filter by 613a38c7,00000000,470b470a,00002ae7 . 

So if the securexl dos module inserted the comment I configured "StdSessionLimit" or the name="DOS-Rate-Limit_Vendor1" instead of the uid, we would be able to filter it.

example: 613a38c7,00000000,470b470a,00002ae7 

fwaccel dos rate get -S fw
operation=add uid=<613a38c7,00000000,470b470a,00002ae7> target=all timeout=none log=alert comment="StdSessionLimit" name="DOS-Rate-Limit_Vendor1"

[Eric_Dale]

Understood.  We will include this feature in a future release.

[Luis_Miguel_Mig]

I would also like to track these alerts in a widget, however I have realized  that the comment field is not available when you create a widget.

[Eric_Dale]

I will learn more about that and see if something can be done

[Eric_Dale]

I spent some time studying the issue. Here's what I have:

1) Source/Destination IP are indexed, but I don't see how to show only DOS/Rate Limiting logs for a give source or destination IP. We are investigating how to make this better.

2) For R80.40, the comments field is only searchable if you are viewing a log file (fw.log). This issue is fixed in R81.00 (PMTR-45132). I have asked the R&D Owner if it can be applied to R80.40 jumbo. It requires a schema change, and I'm not sure if it is easy or hard to apply this sort of schema change to the jumbo

3) I hope to add a query for DOS/Rate Limiting logs, next to the "DDoS Protector" queries in SmartEvent. You didn't ask for this, but I think it makes sense.

4) As already mentioned, we will add the ability to search by name/comment

5) I've never tried the widgets before, but I see what you mean about not having the comment field. I have emailed relevant R&D owner and will discuss with them.

[Egenity]

One additional request, that would make life a LOT easier is:

When configuring the DOS/Rate Limiting, can you please allow single IP address OR ranges to be submitted?  For the ranges, it would be nice to use the following:

xx.xx.xx.xx/nnn

xx.xx.xx.xx/nn.nn.nn.nn

xx.xx.xx.xx-yy.yy.yy.yy

xx.xx.xx.xx yy.yy.yy.yy

Some security services provide IP block lists in varying formats, so automating the entry of this data in a simple way would save a lot of hassle with scripting the conversion.

[Luis_Miguel_Mig]

I am running R81 in the manager and R80.40 in the gateway so I can filter by comment.
By the way, I have just realized a trick to be able to filter just the securexl dos logs.
As I was saying securexl dos inserts the securexl dos rule uid and they all have this format <*,*,*,*>

So filtering by <*,*,*,*> gives you just the securexl dos alerts.

On a separate note, it is a pity that securexl dos rate limit has not visibility of the smartconsole objets, it would be great to be able to use the groups already defined in securexl.
I can understand how securexl dos penalty box, deny list, etc are designed just for emergencies etc and don't require it, however the rate limit function is more than tha. I would like to use it to control the number of concurrent sessions  to specific host anytime non stop and I can't do it with the QOS blade.  It would make it easier if I could use as I said the smartconsoler groups and perhaps the rules already defined in smartconsole or even better if we could configure the securexl dos rate limit rules from smartconsole.

[Eric_Dale]

Regarding adding variations for the range syntax to simplify scripts, that is something that hadn't occurred to me before, but it makes sense.  Separating the two range values with an empty space might be tricky, but it seems doable.  

Regarding SmartConsole object visibility.  I think we would first want to add SmartConsole GUI support for DOS/Rate Limiting.  It has been discussed in the past and would be good to have.  But, it may not happen soon because of other priorities.

[Luis_Miguel_Mig]

I can find those securexl dos logs I created a week ago. It  looks like they have been purged. Is this possible?
I can't find them with blade:Firewall type:alert or <*,*,*,*>

DISREGARD IT.  I opened a new log tab and those logs are back