topic Re: SecureXL DoS Rate Limiting (samp rules) in Firewall and Security Management

SecureXL DoS Rate Limiting (samp rules)

Egenity — Sat, 28 Dec 2019 23:07:08 GMT

I have been working a lot with the rate limiting rules via the "fw samp" CLI interface, but unfortunately I cannot get the gateway to actually enforce them. It appears SecureXL is very unhappy when I try to enable rate limiting:

[Expert@PROD-FW02a:0]# fwaccel dos config set --enable-rate-limit
ERROR: No rate limiting policy is installed, can't enable.

What exactly is the "rate limiting policy" it is referring to?

I have dug fairly deep in documentation, sks, etc. and cannot figure out what triggers the rate limiting capabilities of SecureXL to turn on, based on policy settings. I also thought maybe enabling QoS blade and the QoS policy component would trigger things, but it had no effect on things.

Of course, this same status is reflected when you query the configuration (fwaccel dos config get):

rate limit: disabled (without policy)
pbox: disabled
blacklists: disabled
drop frags: disabled
drop opts: disabledfwacc
internal: disabled
monitor: disabled
log drops: enabled
log pbox: enabled
notif rate: 100 notifications/second
pbox rate: 500 packets/second
pbox tmo: 180 seconds

The gateways are R80.30 5800 appliances.

Re: SecureXL DoS Rate Limiting (samp rules)

Timothy_Hall — Sun, 29 Dec 2019 04:17:44 GMT

Hi Adam,

You need to add a rate-limiting rule first before you can enable enforcement. Please see this article:

https://community.checkpoint.com/t5/General-Management-Topics/How-to-completely-exclude-some-specific-traffic-from-being/m-p/34077

The commands have changed slightly in R80.20+; substitute "fw samp add quota" with "fw sam_policy add quota" and "cat /proc/ppk/dos" with "fwaccel dos stats get". You can also use "fw sam_policy get" to verify a quota rule in R80.20+ after you have added it.

Re: SecureXL DoS Rate Limiting (samp rules)

Egenity — Mon, 30 Dec 2019 13:38:53 GMT

Sorry, I didn't make that totally clear in my initial post. I have a ton of samp rules configured, and verified by fw samp get output. However, it still refuses to kick in.

Re: SecureXL DoS Rate Limiting (samp rules)

Timothy_Hall — Mon, 30 Dec 2019 14:21:26 GMT

By default samp rules will only apply to traffic traversing interfaces defined as External in the firewall topology, have you done a fwaccel dos config set –-enable-internal yet?

Re: SecureXL DoS Rate Limiting (samp rules)

Egenity — Mon, 30 Dec 2019 14:32:35 GMT

Well, in this case, it is really only intended for external interface, so that angle shouldn't matter at this point.

BACKGROUND: Customer wanted to implement an IP block list from a feed. I modified some scripts that Check Point originally supplied in an SK to work properly with the feed and create the SAM rules. That part works perfectly. The gateway just won't actually do anything with them.

Small example of configured SAM policy rules:

[Expert@MDC-PROD-FW02a:0]# fw samp get | more

operation=add uid=<5e09eea8,000020a9,0501010a,0000089f> target=all timeout=1367 action=drop log=log comment=intelligo_ip_block service=any source=range:

1.4.244.35-1.4.244.35 pkt-rate=0 req_type=quota

operation=add uid=<5e09eea8,000020ab,0501010a,0000089f> target=all timeout=1367 action=drop log=log comment=intelligo_ip_block service=any source=range:

1.4.246.250-1.4.246.250 pkt-rate=0 req_type=quota

operation=add uid=<5e09eea8,000020ac,0501010a,0000089f> target=all timeout=1367 action=drop log=log comment=intelligo_ip_block service=any source=range:

Re: SecureXL DoS Rate Limiting (samp rules)

Egenity — Mon, 30 Dec 2019 16:48:05 GMT

After a lot of troubleshooting, it appears to be a size limitation. When I remove the long list and simply configure a test rule, the rate-limiting fires up.

The /var/log/messages gave some clues:

Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_db_rate_rset_rules_alloc (dos_db.c:2838): halloc failed: size=397592
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_db_rate_policy_alloc (dos_db.c:3518): dos_db_rate_rset_alloc failed
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_db_rate_install (dos_db.c:4049): dos_db_rate_policy_alloc failed
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [SIM4];ERROR: [sxl0]dos_q_rate_install (dos_q.c:1257): dos_db_rate_install
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [fw4_0];cphwd_api_q_request_blocking: SecureXL device responded with an error (CPHWD_API_RESPONSE_ERROR). Retry = 0
Dec 30 09:54:00 2019 MDC-PROD-FW02a kernel: [fw4_0];ERROR: vs0: i0: cphwd_dos_ioctl_rate_install_g (cphwd_dos_ioctl.c:422): cphwd_dos_q_request_blocking: sxl_dev_id=0

The block list the customer wants implemented currently has 49,000+ entries (IP ranges).

TAC is a bit perplexed (6-0001867439), and I am not sure they totally understand the issue.

I am curious if there are some adjustments somewhere to accommodate this.

Re: SecureXL DoS Rate Limiting (samp rules)

Timothy_Hall — Mon, 30 Dec 2019 18:24:40 GMT

Looking at your sample rules, it appears you are trying to perform a block with quotas by setting a packet rate of 0 for a single IP address in each statement. The number of quota rules you are trying to install appears to be exceeding some kind of fixed memory/table size in SecureXL, and I don't see any SecureXL kernel variables exposed that could be tweaked to increase the limit.

If you are doing a packet rate of zero to implement a block for all your samp rules, could you perhaps use the new fwaccel dos blacklist command added in R80.20 instead? It is a much simpler feature and may have higher limits for the number of entries you can add.

Re: SecureXL DoS Rate Limiting (samp rules)

HeikoAnkenbrand — Mon, 30 Dec 2019 21:17:36 GMT

Hi @Egenity

to fwaccel dos blacklist read more here: R80.x - Performance Tuning Tip - DDoS „fw sam“ vs. „fwaccel dos“

The SecureXL penalty box is a mechanism that performs an early drop of packets arriving from suspected sources. This mechanism is supported starting in R75.40VS.

Why not sam policy rules?

The SAM policy rules consume some CPU resources on Security Gateway. We recommend to set an expiration that gives you time to investigate, but does not affect performance. The best practice is to keep only the SAM policy rules that you need. If you confirm that an activity is risky, edit the Security Policy, educate users, or otherwise handle the risk. Or better use SecureXL penalty box from a performance point of view.

The purpose of this feature is to allow the Security Gateway to cope better under high load, possibly caused by a DoS/DDoS attack. These commands „fwaccel dos“ and „fwaccel6 dos“ control the Rate Limiting for DoS mitigation techniques in SecureXL on the local security gateway or cluster member.

In version R80.20, the penalty box feature is now supported in VSX mode and each virtual system can be independently configured for penalty box operation.

Attention!

In R80.20, all "sim erdos" commands are no longer supported. They have been replaced with equivalent commands which can be found under "fwaccel dos". Penalty box is configured separately for IPv4 and IPv6. IPv4 configuration is performed using the "fwaccel dos" command. IPv6 configuration is performed using the "fwaccel6 dos" command.

Re: SecureXL DoS Rate Limiting (samp rules)

Egenity — Tue, 31 Dec 2019 21:06:20 GMT

I will examine the blacklist function, but I suspect the same type of limitation may be present. I went with the zero rate limit process, as described in "sk103154 - How to block traffic coming from known malicious IP addresses" -- it appears to be what Check Point recommends.

The other method I tested was a script that builds a dynamic object with all the IP ranges from the black list. It was based on the script and idea from openDBL (discussed in another thread here). Then just referencing the object in the rulebase from drop/log action.

Although, the concept worked, the dyanmic_objects process was a little clunky and performed poorly with 49000+ entries which is why I started down the SecureXL DoS layer implementation. Also, I could never get the dynamic_objects API to pull in all the entries, it would always reject about 5% of the ranges.

Re: SecureXL DoS Rate Limiting (samp rules)

HeikoAnkenbrand — Wed, 01 Jan 2020 20:51:45 GMT

From a performance point of view, I would always use fwaccel dos.

Re: SecureXL DoS Rate Limiting (samp rules)

Eric_Dale — Tue, 07 Jan 2020 15:13:52 GMT

Hi,

A major update to sk112454 was just published. It provides a lot more detail regarding DOS/Rate limiting rules, blacklist, and penalty box. Hopefully it will help.

If you are just trying to block specific source IP addresses, I recommend using fwaccel dos blacklist. Per the sk, there is a hotfix available that will scale the blacklist to millions of IPs. It will be rolled into the R80.20 R80.30 JHF soon.

I can confirm that fw samp rules tend to have resource allocation issues when using more than about 10,000 rules. The root cause is memory allocation failures in the kernel. The work-around is to use the blacklist (create fewer fw samp rules).

Re: SecureXL DoS Rate Limiting (samp rules)

Timothy_Hall — Tue, 07 Jan 2020 15:26:04 GMT

Wow, the updated SK answered all my questions and then some. Great job!

Re: SecureXL DoS Rate Limiting (samp rules)

Egenity — Thu, 05 Mar 2020 14:48:45 GMT

Has the hotfix been rolled in to R80.30 JHF 155? I looked down the list of fixes, but nothing stood out as matching.

Adam

Re: SecureXL DoS Rate Limiting (samp rules)

Eric_Dale — Fri, 06 Mar 2020 12:07:31 GMT

Not delivered yet. It is approved and I expect it will be included in the next ongoing take.

Re: SecureXL DoS Rate Limiting (samp rules)

Egenity — Fri, 27 Mar 2020 17:51:02 GMT

Just to confirm, is the fix in R80.40 release already?

Re: SecureXL DoS Rate Limiting (samp rules)

Eric_Dale — Fri, 27 Mar 2020 18:14:32 GMT

Yes - it is in R80.40 already. Also an easier to use command line for managing fw samp rules. See "fwaccel dos rate --help"

Re: SecureXL DoS Rate Limiting (samp rules)

Luis_Miguel_Mig — Tue, 07 Sep 2021 11:45:51 GMT

Securexl ddos is a great feature and it works really well. I love it.
However it is a pity that the logs generated by securexl DDOS are not indexed and therefore we have not visibility over the feature.
The reality is that it is quite risky to put in production a feature like this without logging/visibility.
When will the logs be indexed? It is very important for a lot of customers I think.

Re: SecureXL DoS Rate Limiting (samp rules)

Eric_Dale — Tue, 07 Sep 2021 14:41:30 GMT

I agree - the logs need to be easily searchable. I know that there is some minimal indexing, but perhaps it is not enough.

Log indexing is outside my area of expertise - if you could provide some details on the limitations you are encountering, and suggestions on what improvements are needed, I will see what can be done.

Re: SecureXL DoS Rate Limiting (samp rules)

Luis_Miguel_Mig — Tue, 07 Sep 2021 16:35:39 GMT

Well I think it is mandatory to be able to to search by source and destination ip for troubleshooting purposes.

But in terms of monitoring we need to be able to identify this type of alerts. The best and easiest way I can think is with the comment and name that fwaccel dos allows you to set with -c and -n.
This way we could totally control the number of fwaccel dos, we could create graphs to track it, etc.

Re: SecureXL DoS Rate Limiting (samp rules)

Eric_Dale — Wed, 08 Sep 2021 12:54:15 GMT

Making logs searchable by comment/name is needed and is already in our list of things to do. I don't have an specific timeframe, but your request should help move it higher in the priority queue.

I confirmed that the logs are already searchable by source/dest IP in SmartConsole. I tested with R81.10 and also R80.20 and it seems to be working.