cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post
Philip_W
Nickel

high cpu allowed but unknown trafic

Hi Checkmates,

For the second week in a row, over the weekend we have been experiencing heavy (allowed) trafic through our VSX (R77.30) toward servers located behind our load balancers. This causes high CPU usage on 2 cores and now we are fearing some targetted DDOS or reconnaissance action is taking place.

We received no complaints from users or server admins. We know which VS is impacted but are having difficulties identifiying exactly what is happening. 

To this end I used:

fw tab -u -t connections | awk '{ print $2 }' | sort -n | uniq -c | sort -nr | head -10

from 'My top 3 CLI commands' ( where Timothy shared a way of showing the top ten source IPs hogging slots in the connection table). This gave us some IPs, but basically we could see that in Smartlog too

How could we investigate this any deeper? Taking a .cap wouldn't help a lot, or would it?

Any ideas?

9 Replies
Highlighted

Re: high cpu allowed but unknown trafic

Hi Philip,

My first course of action would be to run Timothy's super seven commands as found here:

https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac...

In addition, could you answer the following questions please:

1) What are the blades enabled on the affected VSX?

2) Are you aware of any changes that might have taken place two weeks ago?

3) What are the interesting services that are responsible for this increased CPU usage?

Thanks.

 

0 Kudos
Philip_W
Nickel

Re: high cpu allowed but unknown trafic

Hi Nick,

Thanks for pointing me toward Rick Hoppe's s7pac script, I'll run it asap.

1) Enabled blades on that VS: only fw & ips (i know...)

2) Any changes : the environment changes constantly. If anything changes (or if tests are done) this is not always communicated to the network/security guys so it is as good as impossible to find out.

3) Interesting services : we were seeing mainly http/https.

 

0 Kudos

Re: high cpu allowed but unknown trafic

 

According to your description, you know the (DDOS) IP addresses.

I'd block the IP addresses. You can read this article 

R80.x Performance Tuning Tip – DDoS „fw sam“ vs. „fwaccel dos“

On R77:30 ( is out of support in september 2019) use „fw sam“ to block DDOS IP‘s

On R80.10 and above block the DDOS IP on SecureXL layer.

# fwaccel dos blacklist -a <DDOS IP>

 

Tags (1)
0 Kudos

Re: high cpu allowed but unknown trafic

"fw ctl zdebug" is a powertool that is not exhausted from being used with "fw ctl zdebug drop". "fw ctl zdebug" is an R&D tool for testing software in development. Therefore, the insert should be used with care. It starts a debugging in the background until it is aborted with CTRL+C. On productive systems it can have a high performance impact. 
More see here:
 
To debug the DDOS IP use the following command:

fw ctl zdebug +  monitorall | grep -A 40 -B 20 <DDOS IP>

Now you can analyze the session in more detail.

Start Top and take a closer look at the process and cores.

top (+ buttom 1)

Check daemons with high CPU load. More see sk97638 Check Point Processes and Daemons

Check the affinity:

fw ctl affinity -l -a -v

Check enabled blades:

enabled_blades

Check the DDOS IP in the connection table:
fw tab -t connections -f -u | grep <DDOS IP>

Is "aggressive aging" enabled for this IP?


TIP:

Set stricter values in IPS for DDOS attacks.

 

Tags (1)
0 Kudos
Philip_W
Nickel

Re: high cpu allowed but unknown trafic

Hi Guys,

Thanks for the great troubleshooting tips!

0 Kudos

Re: high cpu allowed but unknown trafic

It seems that most responses focus on performance. But is that in effect your greatest concern?

I also sense a request to understand the nature of the traffic. Is that correct?

0 Kudos
Philip_W
Nickel

Re: high cpu allowed but unknown trafic

Sorry, didn't have much time to follow up lately.

 

Yes, indeed Hugo - we were more concerned in detecting exactly what was causing the high CPU. We were thinking of taking a .pcap next time it happens.

0 Kudos

Re: high cpu allowed but unknown trafic

Try using cpview while the high CPU is happening and look at primarily CPU...Top Connections and secondarily Network...Top Connections.  Not sure how this info will be presented in the VSX environment but these screens can usually help identify elephant flows if that is indeed your issue, also take a look at the Connections/sec on the Overview screen in the event the CPU issue is caused by a burst of new connections and subsequent rule base lookup overhead.

 

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com
0 Kudos
Philip_W
Nickel

Re: high cpu allowed but unknown trafic

When the issue reoccured yesterday, we took a .pcap and saw that the load balancers were keeping sending duplicate ACKs between each other. When the session was killed, the problem was solved.

Probably a bug in the loadbalancers...

0 Kudos