Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Philip_W
Contributor

high cpu allowed but unknown trafic

Hi Checkmates,

For the second week in a row, over the weekend we have been experiencing heavy (allowed) trafic through our VSX (R77.30) toward servers located behind our load balancers. This causes high CPU usage on 2 cores and now we are fearing some targetted DDOS or reconnaissance action is taking place.

We received no complaints from users or server admins. We know which VS is impacted but are having difficulties identifiying exactly what is happening. 

To this end I used:

fw tab -u -t connections | awk '{ print $2 }' | sort -n | uniq -c | sort -nr | head -10

from 'My top 3 CLI commands' ( where Timothy shared a way of showing the top ten source IPs hogging slots in the connection table). This gave us some IPs, but basically we could see that in Smartlog too

How could we investigate this any deeper? Taking a .cap wouldn't help a lot, or would it?

Any ideas?

9 Replies
Nick_Doropoulos
Advisor

Hi Philip,

My first course of action would be to run Timothy's super seven commands as found here:

https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac...

In addition, could you answer the following questions please:

1) What are the blades enabled on the affected VSX?

2) Are you aware of any changes that might have taken place two weeks ago?

3) What are the interesting services that are responsible for this increased CPU usage?

Thanks.

 

0 Kudos
Philip_W
Contributor

Hi Nick,

Thanks for pointing me toward Rick Hoppe's s7pac script, I'll run it asap.

1) Enabled blades on that VS: only fw & ips (i know...)

2) Any changes : the environment changes constantly. If anything changes (or if tests are done) this is not always communicated to the network/security guys so it is as good as impossible to find out.

3) Interesting services : we were seeing mainly http/https.

 

0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

 

According to your description, you know the (DDOS) IP addresses.

I'd block the IP addresses. You can read this article 

R80.x Performance Tuning Tip – DDoS „fw sam“ vs. „fwaccel dos“

On R77:30 ( is out of support in september 2019) use „fw sam“ to block DDOS IP‘s

On R80.10 and above block the DDOS IP on SecureXL layer.

# fwaccel dos blacklist -a <DDOS IP>

 

➜ CCSM Elite, CCME, CCTE
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

"fw ctl zdebug" is a powertool that is not exhausted from being used with "fw ctl zdebug drop". "fw ctl zdebug" is an R&D tool for testing software in development. Therefore, the insert should be used with care. It starts a debugging in the background until it is aborted with CTRL+C. On productive systems it can have a high performance impact. 
More see here:
 
To debug the DDOS IP use the following command:

fw ctl zdebug +  monitorall | grep -A 40 -B 20 <DDOS IP>

Now you can analyze the session in more detail.

Start Top and take a closer look at the process and cores.

top (+ buttom 1)

Check daemons with high CPU load. More see sk97638 Check Point Processes and Daemons

Check the affinity:

fw ctl affinity -l -a -v

Check enabled blades:

enabled_blades

Check the DDOS IP in the connection table:
fw tab -t connections -f -u | grep <DDOS IP>

Is "aggressive aging" enabled for this IP?


TIP:

Set stricter values in IPS for DDOS attacks.

 

➜ CCSM Elite, CCME, CCTE
0 Kudos
Philip_W
Contributor

Hi Guys,

Thanks for the great troubleshooting tips!

0 Kudos
Hugo_vd_Kooij
Advisor

It seems that most responses focus on performance. But is that in effect your greatest concern?

I also sense a request to understand the nature of the traffic. Is that correct?

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
Philip_W
Contributor

Sorry, didn't have much time to follow up lately.

 

Yes, indeed Hugo - we were more concerned in detecting exactly what was causing the high CPU. We were thinking of taking a .pcap next time it happens.

0 Kudos
Timothy_Hall
Champion
Champion

Try using cpview while the high CPU is happening and look at primarily CPU...Top Connections and secondarily Network...Top Connections.  Not sure how this info will be presented in the VSX environment but these screens can usually help identify elephant flows if that is indeed your issue, also take a look at the Connections/sec on the Overview screen in the event the CPU issue is caused by a burst of new connections and subsequent rule base lookup overhead.

 

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
Philip_W
Contributor

When the issue reoccured yesterday, we took a .pcap and saw that the load balancers were keeping sending duplicate ACKs between each other. When the session was killed, the problem was solved.

Probably a bug in the loadbalancers...

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events