How to Identify DDoS attack on Check Point Gear

[matanbd]

What is a DDoS Attack?

A distributed denial-of-service (DDoS) attack is a malicious attempt to disrupt normal traffic of a targeted server, service or network by overwhelming the target or its surrounding infrastructure with a flood of Internet traffic. DDoS attacks achieve effectiveness by utilizing multiple compromised computer systems as sources of attack traffic. Exploited machines can include computers and other networked resources such as IoT devices. From a high level, a DDoS attack is like a traffic jam clogging up with highway, preventing regular traffic from arriving at its desired destination.

What is the difference between DoS and DDoS?

Well, nothing except the fact that in DDoS attack the incoming traffic flooding the victim originates from many different sources.
This effectively makes it harder to stop the attack because we cannot simply block a single source.

How Attackers usually launch a DDoS attack

They build an army!
Army of botnets around the world that are waiting for a command.
That command will usually include info such as:

1. Target IP/Domain
2. Type of attack (i.e SYN Flood)
3. Duration/Rate of the attack
4. Time to launch

These botnets could be any type of a computer that is connected to the internet and is currently infected with a piece of malware that allows the attacker to control the device in order to launch the DDoS attack.

Usually when an attacker decides that they want to attack a specific target they won’t start to build an army of botnets for the specific attack.
For that there are already services that ran by hackers which offer the use of their already prepared infrastructure of botnets.

A nice example is of two young Israeli hackers that were able to compromise millions of IoT devices all around the world and used them to launch DDoS attack on specific targets.
You can read more about it here:
https://krebsonsecurity.com/2016/09/israeli-online-attack-service-vdos-earned-600000-in-two-years/

How to prove/disprove DDoS on CheckPoint Firewall

The easiest and quickest way to prove or disprove if a DDoS attack occurred on one of CheckPoint’s Firewalls, would be to use cpview tool to go back in time to the point of the attack and look for a spike in the CPU utilization and packet rate.

If you are not familiar with cpview tool, check out sk101878 

In version R77.10, cpview got the History Mode feature that automatically keeps all cpview’s records for 30 days backwards.
With this feature we can go back in time to the point of the suspected DDoS attack and check what exactly happened during this time on the GW.
This can be done by running cpview with the following syntax:

# cpview -t <timestamp>

where <timestamp> format is: [Jan...Dec] [01...31] [4-digit Year] [hh:mm:ss]

IMPORTANT NOTE: Starting R80.30 the syntax of CPView history mode has changed, this guide was generated using older versions and the old syntax. please review sk163804 for the new syntax.

For example, if you believe a DDoS attack occurred on April 30, 2019 during 13:00 – 15:00 we will execute cpview with the following timestamp:

# cpview -t Apr 30 2019 12:30:00

Note that we set the timeframe to 30 minutes prior to when the attack started, this will later help us compare between during the attack and normal operation.

CPVIEW Overview#1 – CPU utilization section
#2 – Network section

We need to keep in mind the following values of a normal operation:
CPU utilization
Packets/sec
Now using the ‘+’ character we will go forward in time, minute-by-minute and keep monitoring these values.
We will look for a massive increase or spikes.
In case we are dealing with a DDoS attack the CPU utilization and packets per second rate should be correlated to each other.

Scenario 1 – CPU utilization increasing, packet/sec remain the same

In case we only see the CPU utilization rise during the mentioned time frame, it means that we are most likely not dealing with a DDoS attack and instead we might be facing a performance issue on the GW that should be investigated.

Solution

Technical Support (TAC) should investigate performance issue.

Scenario 2 – Packets/sec increasing, CPU levels are OK (below 80%)

In case we see an increase of packets/sec value without CPU levels increasing, it could indicate a DDoS attack that our GW is not affected by and in that case it could be that the end device is not capable of handling the traffic.

The next step should be to find out which traffic is causing such behavior, it can be done with the following actions:

1. In cpview go to Network --> Top Connections/Protocols
   if there is data in these sections, verify with customer that the top connections indeed shows an external IP/s that customer is no familiar with (could be an internal server generating the traffic)
   Note: these sections in cpview might not contain any data, in this case continue to the next option.
2. Review FW logs for the time of the incident by setting Custom Time:
   SmartView time frame filter
3. Click on “Top Sources” in the right “Tops” view:
   SmartView Top Sources view

Solution

1. If possible implement one of the following:
   • Add SAM rule to block the traffic based on traffic's patterns.
   • Add FW rule (closer to the top of the rulebase) to block the malicious traffic
2. Refer customer to Incident Response Team if they wish to continue the investigation or if the above actions did not resolved the issue.

Scenario 3 – Packets/sec increasing, CPU levels increasing accordingly

In this scenario we are most likely dealing with a DDoS attack, but same as scenario #2, we need to let the customer review the source of the traffic to make sure it’s not an internal server or other source that they are familiar with.
Use steps from scenario #2 to verify with the customer.

Solution

1. If possible implement one of the following:
   • Add SAM rule to block the traffic based on traffic's patterns.
   • Add FW rule (close to the top of the rulebase) to block the malicious traffic
2. Refer customer to Incident Response Team if they wish to continue the investigation or if the above actions did not resolved the issue

CPView in VSX mode

In VSX mode, cpview is supported only from the context of VS 0.
The network section in the main view may show us irrelevant data, the CPU utilization section will show us the correct information.
The approach in this case is to view the Packets/sec value on the interface level instead of the summary of all interfaces.

 To do so we need to verify which interface is experiencing the DDoS (usually the internet facing interface), and then go to Network --> Interfaces --> Traffic:
CPVIEW Interfaces traffic view

In the RX Traffic (Received) section, we should focus on the “pps” column and look for the spike/increase.

How to determine DDoS in a packet capture

In case you suspect a DDoS attack and a traffic capture from the time of the attack has been collected, the following will describe how we should determine if the packet capture contains any signs of a DDoS attack

Let’s review a few general key points that we should look for inside the traffic capture file:

1. Comparing the count of SYN packets with SYN-ACK packets can help detect a SYN flood attack, in a normal packet capture the count between the two should be close to equal (for valid connections, every SYN should have a corresponding SYN-ACK)
2. Need to compare traffic volume between normal state and the time of DDoS.
   This can be done by comparing the Packets Per Second (PPS) between the two traffic captures.
3. Need to verify traffic directions: external ---> Internal, Internal ---> External, Internal --> Internal
4. How much traffic in size was sent to the destination?
5. Need to verify traffic destination – multiple or one destination? Which type of server? Is it logical that this server will be attacked?
6. Internal resource is active?

All of the above key points seem hard to answer, but luckily Check Point has a tool that can help us answer all of them.
The tool name is CPMonitor and can be found inside sk103212

CPMonitor can work in two different modes:

1. Complete Mode – provides statistics and summary of the entire traffic capture
2. Navigate Mode – allows you to navigate minute by minute through the traffic capture while showing relevant statistics for each minute.

When investigating a suspected DDoS attack inside a packet capture file, we should start with the “Complete Mode” in order to verify points 1-4 from the above list.

The command to run is:

# cpmonitor /Path_To_pcap_File/traffic.pcap

The output should like this:

=====================================
| Dump Info |
=====================================
| Name: SYN_Defender_on_April_11.pcap |
| Start time: 04/12/19 02:34:35.41 |
| End time: 04/12/19 02:36:43.10 |
=====================================
=================================================================
| Total Usage |
=================================================================
| Capture duration: 127.688 seconds |
| Total packets: 18273340 |
| Total packets size: 9660 MB |
| Avg. PPS: 143109.30 [packets/second] |
| Max PPS: 151940 [packets/second] at 02:35:21.41 |
| Avg. CPS: 4231.74 [conns/second] |
| Max CPS: 10530 [conns/second] at 02:34:36.41 |
| Avg. throughput: 605 Mb [bits/second] |
| Max throughput: 663 Mb [bits/second] at 02:35:30.41 |
| Avg. concurrent conns: 17010.09 [conns/second] |
| Max concurrent conns: 17576 [conns/second] at 02:35:01.41 |
| Sort method: packets rate |
| Unsupported entries: 827 |
=================================================================
Top Connections
========================================================================================================================
| source | port | dest | port | ipp | packets (%) [c2s/s2c] | size (%) [c2s/s2c] | avg | desc. |
========================================================================================================================
| 10.24.129.44 | 39375 | 157.240.14.52 | 443 | TCP | 49246 (0%) [45%/55%] | 27 MB (0%) [3%/97%] | 42/1038 | https |
| 10.26.187.252 | 43638 | 200.7.241.12 | 443 | UDP | 25748 (0%) [32%/68%] | 23 MB (0%) [2%/98%] | 68/1376 | https |
| 10.26.215.106 | 54760 | 200.7.241.13 | 80 | TCP | 24960 (0%) [24%/76%] | 26 MB (0%) [0%/100%] | 42/1477 | http |
| 10.24.107.79 | 37846 | 190.98.133.161 | 443 | TCP | 22769 (0%) [43%/57%] | 13 MB (0%) [3%/97%] | 42/1045 | https |
| 10.197.229.143 | 57768 | 200.7.241.14 | 443 | TCP | 22001 (0%) [35%/65%] | 20 MB (0%) [1%/99%] | 42/1472 | https |
| 10.24.212.89 | 60172 | 200.7.241.13 | 443 | TCP | 17848 (0%) [33%/67%] | 12 MB (0%) [1%/99%] | 42/1049 | https |
| 10.24.248.44 | 54129 | 157.240.14.52 | 443 | TCP | 16619 (0%) [63%/37%] | 5919 KB (0%) [95%/5%] | 552/40 | https |
| 10.24.128.188 | 34548 | 200.7.241.12 | 443 | TCP | 16331 (0%) [32%/68%] | 11 MB (0%) [1%/99%] | 42/1049 | https |
| 10.26.195.81 | 49283 | 200.7.241.13 | 443 | TCP | 16040 (0%) [32%/68%] | 11 MB (0%) [1%/99%] | 42/1051 | https |
| 10.209.49.116 | 43608 | 200.7.241.12 | 443 | TCP | 15915 (0%) [33%/67%] | 10 MB (0%) [1%/99%] | 42/1050 | https |
========================================================================================================================
Top destinations
==========================================================================
| IP | packets (%) [c2s/s2c] | size (%) [c2s/s2c] | avg |
==========================================================================
| 157.240.14.52 | 812157 (4%) [51%/49%] | 418 MB (4%) [4%/96%] | 357/733 |
| 190.98.133.225 | 759828 (4%) [39%/61%] | 519 MB (5%) [2%/98%] | 50/1149 |
| 157.240.14.15 | 731021 (4%) [49%/51%] | 260 MB (2%) [15%/85%] | 231/514 |
| 200.7.241.12 | 626071 (3%) [35%/65%] | 507 MB (5%) [2%/98%] | 62/1283 |
| 200.7.241.13 | 610489 (3%) [34%/66%] | 485 MB (5%) [2%/98%] | 56/1242 |
| 168.195.56.82 | 600580 (3%) [38%/62%] | 440 MB (4%) [2%/98%] | 53/1212 |
| 200.7.241.14 | 549053 (3%) [32%/68%] | 475 MB (4%) [2%/98%] | 66/1306 |
| 190.98.133.161 | 533195 (2%) [38%/62%] | 368 MB (3%) [2%/98%] | 49/1152 |
| 157.240.14.53 | 484829 (2%) [49%/51%] | 68 MB (0%) [32%/68%] | 96/199 |
| 168.195.56.97 | 445217 (2%) [38%/62%] | 296 MB (3%) [3%/97%] | 55/1103 |
==========================================================================
Top Services
==============================================================================================
| service | ipp | packets (%) [c2s/s2c] | size (%) [c2s/s2c] | avg | desc. |
==============================================================================================
| 443 | TCP | 13898938 (76%) [42%/58%] | 7681 MB (79%) [0%/100%] | 129/915 | https |
| 443 | UDP | 1300254 (7%) [34%/66%] | 1105 MB (11%) [1%/99%] | 234/1229 | https |
| 80 | TCP | 783018 (4%) [43%/57%] | 440 MB (4%) [6%/94%] | 88/978 | http |
| 53 | UDP | 477804 (2%) [51%/49%] | 46 MB (0%) [33%/67%] | 67/140 | DNS |
| 5222 | TCP | 400813 (2%) [48%/52%] | 54 MB (0%) [32%/68%] | 93/187 | xmpp-client |
| 3478 | UDP | 211739 (1%) [45%/55%] | 85 MB (0%) [0%/100%] | 450/398 | stun |
| 7008 | UDP | 135954 (0%) [25%/75%] | 8907 KB (0%) [24%/76%] | 64/67 | afs3-update |
| 5228 | TCP | 128290 (0%) [44%/56%] | 36 MB (0%) [14%/86%] | 94/456 | hpvroom |
| 10004 | UDP | 80584 (0%) [32%/68%] | 7393 KB (0%) [20%/80%] | 59/110 | emcrmirccd |
| 10003 | UDP | 56965 (0%) [27%/73%] | 5067 KB (0%) [17%/83%] | 57/104 | documentum-s |
==============================================================================================
=============================
| TCP Stat | Count |
=============================
| SYN | 293174 |
| SYN|ACK | 290635 |
| SYN|ACK|PUSH | 2 |
| SYN|PUSH | 2 |
| ACK | 12327501 |
| ACK|PUSH | 2053247 |
| ACK|URG|PUSH | 1 |
| FIN|ACK | 216722 |
| FIN|ACK|PUSH | 59617 |
| FIN|ACK|URG | 1 |
| FIN|ACK|URG|PUSH | 1 |
| RST | 219186 |
| RST|ACK | 20819 |
| Invalid | 25 |
=============================
 

IMPORTANT NOTE: Traffic capture should be taken without any filters. Most of the tests we do depend on anomaly detection and therefore we need the DDoS traffic to “stand out” from the rest of the traffic.

 Key point #1

CPMonitor TCP State Count

As we can see in the above example, the number of SYN-ACKs is almost the same as the number of total SYNs (no DDoS this time).
During SYN Flood DDoS attack the server in question won’t be able to answer all the SYN requests that are coming towards the server due to the high volume of requests (DDoS).

Key point #2

CPMonitor Total Usage

The important part is the “Avg. PPS:” value, should be significantly higher when comparing the traffic captures from the time of DDoS and the normal operation time.

Key points #3 & #4

CPMonitor Top Connections

In this section we can detect the top connections directions, it is very important to identify the directions for the top connections that are involved in the DDoS attack so we can determine if this is caused by an internal/external resource.

In addition using this section we can also tell how much traffic in size was sent by the server and how much was sent by the client, see “size (%) [c2s/s2c]” column.
When dealing with normal traffic, the server is usually the one sending larger volumes of traffic to the client and not the other way around (except for uploads and probably other edge cases).
If we notice the client is sending more traffic than the server, this might indicate a DDoS attack.

Key points #5 & #6

Key points #5 & #6 can be detected by running CPMonitor in “Navigate” mode.
Using this mode we can go through the traffic capture second by second and see how the attack behaves, we can still review the above mentioned point while running in Navigate mode. It’s important to look for the destination IPs that are being communicated the most and in case this is an internal resource like a web server for example, we need to make sure that it is active and responding (at least to some of the requests).

One of the most common scenarios in which customer claim there is a DDoS attack when there isn't, is when an internal resource like a web server becomes unavailable/not responding and then all the clients are trying to connect/reconnect at the same time.

In the traffic logs we see many attempts to open new connections to the website and it may look like a DDoS attack, but if we see that the web server is not replying to any of the requests at all it means that these are probably legitimate requests to the website which is not responding.

Final Notes & Recommendations

1. The above guide does not cover a scenario in which an application has a DDoS vulnerability that can be exploited, this should be handled by IPS blade and if not the application traffic should be investigated separately, CPIRT would be happy to assist in this case.
2. sk112241 - Best Practices - DDoS attacks on Check Point Security Gateway is a good guide for tuning your GW's performance to allow more traffic to be handled.
3. if you your gateway is constantly utilizing 70%-90% of it's CPU, it is recommended to approach TAC and request to do performance optimization on your gateway to allow him to handle more traffic and better prepare for the next DDoS attack.
4. Generally speaking, Check Point mainline products are not design to mitigate DDoS attacks, there are dedicated products for mitigating DDoS attacks such as DDoS protector appliance and DNS Cloud Scrubbing services that can filter DDoS attack before they enter your network. CPIRT always recommends using one of those for a permanent solution.
5. if you need help with any type of cyber incident, Check Point's Incident Response Team is always happy to assist, our contact numbers & email is available Here
   
   
   For the full list of White Papers, go here. 
   
   

[Shay_Levin]

@matanbd This is an awesome document!  Everyone should read it.

For those of you who don't know Matan, he is one of the most talented Check Point incident response team member.

So, if you are in trouble, Matan is definitely your guy.

[Matthias_Kring]

With 80.30, the "timestamp" in cpview -t seems to be unavailable (sk101878). Instead, it starts at the beginning of history, and you have to move forward in "minute steps".

That's not nice.

[matanbd]

@Matthias_Kring Hi Matthias,

thanks for letting me know about this.

it looks like the design has been changed since R80.30 in CPView history mode and sk163804 describe the change:

1. Run the 'cpview -t' command.
2. In CPView history, type t.
3. Specify the desired date and time. 

i will add this info to the post itself.

BR
Matan

[pahanadmin]

Great,what about SMB devices..

[jfran3]

Is there any guidance for detecting this using only the SmartConnect interface? So if we have analysts who don't have access to run cpview for instance?