Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
JulzorenSen
Contributor
Jump to solution

Check Point Scans/DoS protection is BROKEN

Hello everyone, first of all excuse me for click-bait title - stay with me!

We are long-time Check Point users and are pretty happy with the solutions they provide.

However, this specific point bothers us a lot, especially my management.

 

Some Background

We are a big institution (50k users) and at the edge (internet facing) we use a cluster of CP 16000-T with all the Next-Gen blade activated (currently in R80.30 last JHF).

The specitify that we have compared to some other clients might be that we own, for historical reasons, a full /16 of IPv4 public address (+ some other ranges aswel), and also a /29 of public IPv6 address (we are considered a Local Internet Registry depending on RIPE-NCC).

As everyone, we are constanly scanned by a sh**load of bots in the world. But with that much public IPs, this can represent a HUGE number of con/sec and thus might as some point be considered as small (D)DoS, especially for internal firewalls (about ~50) which are way smaller.

In addition of security concerns that it repesents, it can actually break stuff.

So obviously, this kind of basic attacks/scans/dos are expected to be blocked right at the edge by our 16000-T cluster.

The main type of attack concerned here is "Sweep Scans".

 

The issue

Before using Check Point at the edge, we had some Palo Alto firewalls for 10 years that were doing this job perfectly (with the Zone Protection feature).

We also had at some point some Fortinet firewalls which were also doing the job (DDoS policies).

However, with Check Point, it looks like to me that is simply impossible to block these type of scans.

 

I already did post a message 2 years back with the details of the investigation at this time, but I have new information since then : https://community.checkpoint.com/t5/Threat-Prevention/IPS-Sweep-Scan-issue-a-LOT-of-scans-are-not-be...

 

First of all, the IPS signature for Sweep Scans can only be configured in "Detect" Mode, not Prevent.

There is an SK (sk110873) explaining how handle this by configuring custom User Alert combined with sam_alert to generate a SAM rule and block the source IP of a given scan.

To further improve this, we did configure the Penalty Box for this malicious source IP to be dropped ASAP and not consume unnecessary ressources on the gateways.

 

Looks complicated compared to the other major players (PA, Forti), but whatever : looks good for us.

Except that... it doesn't work.

 

Some scans were blocked correctly using this method, but the vast majority were still accepted and reaching the internal network.

 

A case was of course opened : 6-0001785477

The case was handled correctly, we had a lot of contact with R&D, but the conclusion was the Good Old "it's working by design" (not blaming, everyone does that ;-)).

 

Here is what R&D told us after some code reviews :

[...]

Following R&D investigation and testing the 'Sweep Scan' protection, The Design is as follows: 

1.            The protection is not designed for blocking Sweep Scan malicious IPs. It has the setting of 'detect / accept' or 'inactive'. 
2.            The protection will count attempts for certain port only until it reaches the threshold defined in the protection. After that, Until the timeout will expire it will not proceed counting additional sources attempting to attack same port (even if different destinations). 
3.            The protection will count each source reaching 5% or above the defined threshold, until the accumulated threshold has been reached. After that, Until the timeout will expire it will not proceed counting additional sources attempting to attack same port (even if different destinations). 
4.            Additional sources from point 2 will be displayed in the ‘more sources’ field in the logs.

[...]

 

The important thing to note here is point n° 4.

If multiple sources do scans on the same dst.port, only the first one will get his "own log entry", thus triggering the custom User Alert, then triggering the SAM Rule and ultimately ending in the Penalty Box.

The following sources are simply being added in the same log entry under a "More sources" field, but these ones will never generate any alert/sam_rules and will never get blocked.

 

In our case, this is a huge issue.

The scans are sometimes that huge that it might break things, and by not being blocked properly (for example by ending in the Penalty Box) it basically consumes up to 20% on the 16000-T CPUs (!!!).

I'am pretty sure if it was handled correctly that wouldn't even be noticed (again please remember that this is working perfectly with all the other major players).

 

An RFE was opened back at that time, but nothing moved since then.

I'am still being told that I'am the only client to ask for such feature (again, I'am used to hear that with every vendors... :-)).

How could it be possible that no one faces that?

 

The conclusion

So what we had to do to protect the edge Check Point gateways (sad to say that this way...) is installing a Snort and spanning all the edge trafic to it.

When the Snort detects too much scanning from a source IP it blocks it on some Cisco edge router (with some scripts to maintain this list correctly).

 

My management is asking me time to time why we have to maintain an Open Source Snort after spending triple digits thousands € on a pro' solution. For once, I'am kind of aligned with what they ask.

They are of course very aware that we didn't have a single issue regarding this for the 15 years before using Check Point NGFWs, neither with PA nor Forti.

 

So, what are you all thinking about that?

I'am especially interested in the forums experts and Check Point interns point of view.

 

Don't you think that Check Point firewalls misses a piece of Unified-Infinity-EasyToUse protection regarding Scans and DoS?

Every other vendors does support <10clicks config to handle that : PA, Forti, Cisco, and even some Open Source software like Snort.

 

Is there ANY possibility that I could have missed to handle this correctly on my gateways?

 

Is there any improvements coming regarding this?

 

Again, I might look angry and sarcastic but I'am NOT (well, maybe just a bit 😋).

I'am really doing that in an improvement optic (either on my side or CP's side).

2 Solutions

Accepted Solutions
Wolfgang
Authority
Authority

@JulzorenSen how about using SAM rules created via SmartEvent? We are using these a long time with success.

Screenshot 2021-04-29 162148.png

 

 

Wolfgang

View solution in original post

frankcar
Contributor

this tells you what you want.

 

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_CLI_ReferenceGuide/Topics-CLIG/FWG...

 

 

  • est Practice - The SAM Policy rules consume some CPU resources on Security Gateway. Set an expiration for rules that gives you time to investigate, but does not affect performance. Keep only the required SAM Policy rules. If you confirm that an activity is risky, edit the Security Policy, educate users, or otherwise handle the risk.

  • Logs for enforced SAM rules (configured with the fw sam command) are stored in the $FWDIR/log/sam.dat file.

    By design, the file is purged when the number of stored entries reaches 100,000.

    This data log file contains the records in one of these formats:

    <type>,<actions>,<expire>,<ipaddr>
    <type>,<actions>,<expire>,<src>,<dst>,<dport>,<ip_p>
  • SAM Requests are stored on the Security Gateway in the kernel table sam_requests.

  • IP Addresses that are blocked by SAM rules, are stored on the Security Gateway in the kernel table sam_blocked_ips.

 

 

 

View solution in original post

(1)
24 Replies
JulzorenSen
Contributor

Forgot to mention something important to understand our issue.

 

Our internet facing firewall are in full "Accept" mode, they are only enforcing Next-Gen stuff (URL Filtering, IPS, Anti-Bot, etc.).

That's why these kind of scans are flowing trough the Intranet and should be blocked at the IPS level.

 

Granular rules (with implicit Deny)  are handled by ~50 gateways (in legacy mode - only Firewall Blade enabled) inside the network.

0 Kudos
Nik_Bloemers
Advisor

If I may hazard a guess, I suppose Check Point assumes you'll buy something like their DDoS protection solution for this sort of job perhaps? So it might indeed be by design in that regard.

https://www.checkpoint.com/quantum/ddos-protector/

JulzorenSen
Contributor

Thank you for you guess, however I hope that it is false 😉

I'am no talking about volumetric DDoS or some kind of super-smart-attacks here.

Even if in our case, due to the number of public IPs that we own, it turns out to be critical, it's still just about simple Sweep Scans.

We're not trying to do Arbor or cloud-based anti-DDoS solutions stuff with our CP gateways, just to block some basics recon' attacks, which a firewall should be able to do (and all of them does except Check Point, until now).

In addition, the gateway looks like it can detect these Scans, but there is no way to block them properly, which is kind of sad.

0 Kudos
the_rock
Legend
Legend

Wow, thats fantastic explanation you provided, by far the best I had seen so far on any port, so big kudos to you. Please forgive me for my ignorance if I misunderstood what you are looking for, but to make long story short, you simply want to make sure ddos scanning/attacks would be blocked via IPS? Or did I miss that completelly?

0 Kudos
the_rock
Legend
Legend

Interesting sk I found for this on support site is sk112241. Ever checked that one out?

0 Kudos
JulzorenSen
Contributor

Hehe thank you man, much appreciated 😁

 

What we want : as you stated, we just want IPS Sweep Scan to work and get blocked by our gateways (either for IPV4, IPV6, UDP, TCP, ICMP, ...). 

If a source IP try to scan more than X destination IPs on my network in X seconds, blacklist this source IP for X minutes.

 

About sk112241, we did actually implement almost everything in this list.

About SYNATK, this is more a global feature as it doesn't make any distinction of the source IP as soon as it is enforcing. This might also break some legitimate stuff (and it does), so it's more a "Last Resort" feature in my opinion (with much higher threshold, to cover real DDoS attacks).

Also, it obviously only work for TCP.

Aggressive Aging is enabled, but it doesn't prevent the trafic to enter the network.

Network Quota might be the closest for what we are looking for, however it doesn't make any distinction if multiple destinations are targeted, so if a legitimate app' need to make/receive a lot of con/sec, even to a single destination, it might get blocked (which means we would need to maitain a list of exceptions, potentially huge). Also, I saw a lot of post stating that this feature should be avoided since it disables acceleration and is CPU intensive.

Rate Limiting Rules : I didn't try this but it's off topic for similar reasons as Network Quota is. I might still have a look at it for my culture.

The other points are mainly implemented but irrelevant here.

 

Feel free to correct me if I'am wrong on anything here 😉

 

EDIT : where is my Kudo??!! 😊

the_rock
Legend
Legend

Be free to message me privately, lets discuss.

0 Kudos
Wolfgang
Authority
Authority

@JulzorenSen how about using SAM rules created via SmartEvent? We are using these a long time with success.

Screenshot 2021-04-29 162148.png

 

 

Wolfgang

the_rock
Legend
Legend

Hm, that actually looks very logical to me. @JulzorenSen , ever attempted to do what Wolfgang suggested?

0 Kudos
JulzorenSen
Contributor

Thank you for your input.

To be perfectly honest, we do not use SmartEvent since we have our own SIEM, it is not activated right now.

However, I was aware of these functionalities and I'am almost sure that we tested that before.

I will re-activate it asap to test this, just to be sure.

Can you confirm that you can see multiple sources scanning on the same port getting blocked?

I'll update the post accordingly, but I might need a little bit of time to make the changes here! 😉

0 Kudos
Wolfgang
Authority
Authority

Yes, I'm sure. But I think it's hard to find the correct values (x connections in y seconds) for your environment.

I' don't know your environment and requirements, but I would prefer to block most of the traffic as early as possible, not let them through the first system and blocking them later after running through my internal network.

 

Description of "IP sweep"-Scan from SmartEvent.

IP sweep is an excessive number of attempts to scan the internal network in order to discover hosts or servers that can be accessed through a specific service. The scan is performed by a specific machine from the external network that uses the same protocol and service with each attempted connection.
Once the number of unique destination IP addresses exceeds the defined threshold and at least one connection passes the firewall, this event will appear.

JulzorenSen
Contributor

Alright, thank you 🙂 

I'll try with the same values that the Snort is enforcing right now : 200 dst in 10 sec and block the source for 10 minutes.

The goal here is to mainly protect the Network&Sec infrastructure, but to avoid any (or a maximum of) false positives. We'll maybe tweak it afterwards. 🙂

I'll keep you updated.

JulzorenSen
Contributor

Hello,

Here is the update!

I didn't know that SmartEvent licensing changed toward subscription with R8x, so... I don't have a license anymore 😉 and since we don't really use it, we probably won't get any.

However, the Correlation Unit seems to work even without a licence, so I could try.

 

It looks like it works!!! I configured the Sweep Scans threshold and installed the Event Policy, then I immediatly saw some "Correlated" Logs.

However, the SAM rules didn't get installed (sam file size exceeded error message).

But every source IP get his own log line, so I guess it's supposed to work, I'll dig more and see what I can do.

 

Any idea about the SAM error message?

 

Anyway, it looks like I missed this SmartEvent things in my quest, so NO kudos for me 😉

NO kudos for Check Point either, for never pointing me towards this direction, even after hours of talking about this subject! 😉

0 Kudos
Wolfgang
Authority
Authority

A screenshot or detailed error message would be helpful to analyze the SAM rule problem.

Wolfgang

0 Kudos
frankcar
Contributor

we had this with Sam file error

try

sk97306

set 6 onwards.

G_W_Albrecht
Legend
Legend

sk97306 sayz: Fixed in Check Point R77.20. Who still uses Check Point R77.10 or lower ?

CCSE CCTE CCSM SMB Specialist
0 Kudos
frankcar
Contributor

typo, meant step 6 on,

we on R80.40, the tick box is not on by default, 

we got some older version running real pain to restore if RMA'd

we used this protection, sk110873 shame it does not mention to set this tick box in this SK.

also using smartevent to monitor actions, and have a report that generated on host-scan ips events.

pddos is also enabled for a few hosts, need to upgrade to get better logs for it R81.10 index them older version dont so its not easy to find in logs.

 

 

0 Kudos
G_W_Albrecht
Legend
Legend

I understand, you mean:

Expand 'Other' - go to 'SAM' pane - check the box 'Purge SAM file when it reaches:' - set the desired limit - click in 'OK'. 

CCSE CCTE CCSM SMB Specialist
0 Kudos
Lukas_Sosnovec
Contributor
Contributor

Hi all,

sorry to reopening a 'solved' discussion, I would like to dig a bit deeper here. Does anybody know what exactly will "Purge SAM file when it reaches" checkbox do? We are using SmartEvent for the purpose described before - to block scans, IPs which reached defined rate limit and so on. In work well, except that sometimes I got this SAM file size error.

Now I am trying to understand how exactly does the sam.dat work. 
- sk97306 says 'The FWD module stores the Suspicious Activity Monitoring (SAM) rules received from the FWM module in the $FWDIR/log/sam.dat file'.
- But admin guide says 'Logs for enforced SAM rules (configured with the fw sam command) are stored in the $FWDIR/log/sam.dat
- sk97306  also notes "The SAM records file contains all requests sent to the Security Gateway including obsolete requests."

So what exactly is stored in the dat file? Logs? SAM rules? What are those obsolete requests? Why are they obsolete?
If I enable the workaroung from sk97306 what exactly will be purged if my sam.dat file excedes the specified limit? Is there any danger that active SAM rules will be deleted?

0 Kudos
G_W_Albrecht
Legend
Legend

sk97306 is from 2015 and covers an issue fixed in R77.20, where the sam.dat file will not be purged correctly from obsolete information. So you will not use sk97306 ever, why the question ?

CCSE CCTE CCSM SMB Specialist
0 Kudos
Lukas_Sosnovec
Contributor
Contributor

well, if I understand sk97306 correctly, it introduced 'Purge SAM file when it reaches:' in R77.20 as a hotfix. And this mechanism is still present in current versions.

You can follow sk97306  from step 6 on. It really works, I set the limit to 1M and the size of sam.dat really dropped to 1M. What I am trying to understand now is what exactly was deleted from this file? CheckPoint documentation is incosistent on this toppic.

0 Kudos
PhoneBoy
Admin
Admin

I presume it's just the SAM rules.

G_W_Albrecht
Legend
Legend

That is stated clearly in sk97306: obsolete information ! Fixes an issue that prevents the purge of the history file.

If you need the detailed information for a customer, ask TAC or the local CP SE !

CCSE CCTE CCSM SMB Specialist
frankcar
Contributor

this tells you what you want.

 

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_CLI_ReferenceGuide/Topics-CLIG/FWG...

 

 

  • est Practice - The SAM Policy rules consume some CPU resources on Security Gateway. Set an expiration for rules that gives you time to investigate, but does not affect performance. Keep only the required SAM Policy rules. If you confirm that an activity is risky, edit the Security Policy, educate users, or otherwise handle the risk.

  • Logs for enforced SAM rules (configured with the fw sam command) are stored in the $FWDIR/log/sam.dat file.

    By design, the file is purged when the number of stored entries reaches 100,000.

    This data log file contains the records in one of these formats:

    <type>,<actions>,<expire>,<ipaddr>
    <type>,<actions>,<expire>,<src>,<dst>,<dport>,<ip_p>
  • SAM Requests are stored on the Security Gateway in the kernel table sam_requests.

  • IP Addresses that are blocked by SAM rules, are stored on the Security Gateway in the kernel table sam_blocked_ips.

 

 

 

(1)

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events