high cpu allowed but unknown trafic

[Philip_W]

Hi Checkmates,

For the second week in a row, over the weekend we have been experiencing heavy (allowed) trafic through our VSX (R77.30) toward servers located behind our load balancers. This causes high CPU usage on 2 cores and now we are fearing some targetted DDOS or reconnaissance action is taking place.

We received no complaints from users or server admins. We know which VS is impacted but are having difficulties identifiying exactly what is happening. 

To this end I used:

fw tab -u -t connections | awk '{ print $2 }' | sort -n | uniq -c | sort -nr | head -10

from 'My top 3 CLI commands' ( where Timothy shared a way of showing the top ten source IPs hogging slots in the connection table). This gave us some IPs, but basically we could see that in Smartlog too

How could we investigate this any deeper? Taking a .cap wouldn't help a lot, or would it?

Any ideas?

[Nick_Doropoulos]

Hi Philip,

My first course of action would be to run Timothy's super seven commands as found here:

https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac...

In addition, could you answer the following questions please:

1) What are the blades enabled on the affected VSX?

2) Are you aware of any changes that might have taken place two weeks ago?

3) What are the interesting services that are responsible for this increased CPU usage?

Thanks.

[Philip_W]

Hi Nick,

Thanks for pointing me toward Rick Hoppe's s7pac script, I'll run it asap.

1) Enabled blades on that VS: only fw & ips (i know...)

2) Any changes : the environment changes constantly. If anything changes (or if tests are done) this is not always communicated to the network/security guys so it is as good as impossible to find out.

3) Interesting services : we were seeing mainly http/https.

[HeikoAnkenbrand]

According to your description, you know the (DDOS) IP addresses.

I'd block the IP addresses. You can read this article 

R80.x Performance Tuning Tip – DDoS „fw sam“ vs. „fwaccel dos“

On R77:30 ( is out of support in september 2019) use „fw sam“ to block DDOS IP‘s

On R80.10 and above block the DDOS IP on SecureXL layer.

# fwaccel dos blacklist -a <DDOS IP>

[HeikoAnkenbrand]

"fw ctl zdebug" is a powertool that is not exhausted from being used with "fw ctl zdebug drop". "fw ctl zdebug" is an R&D tool for testing software in development. Therefore, the insert should be used with care. It starts a debugging in the background until it is aborted with CTRL+C. On productive systems it can have a high performance impact. 

More see here:

"fw ctl zdebug" Helpful Command Combinations

To debug the DDOS IP use the following command:

fw ctl zdebug +  monitorall | grep -A 40 -B 20 <DDOS IP>

Now you can analyze the session in more detail.

Start Top and take a closer look at the process and cores.

top (+ buttom 1)

Check daemons with high CPU load. More see sk97638 Check Point Processes and Daemons

Check the affinity:

fw ctl affinity -l -a -v

Check enabled blades:

enabled_blades

Check the DDOS IP in the connection table:
fw tab -t connections -f -u | grep <DDOS IP>

Is "aggressive aging" enabled for this IP?

TIP:

Set stricter values in IPS for DDOS attacks.

[Philip_W]

Hi Guys,

Thanks for the great troubleshooting tips!

[Hugo_vd_Kooij]

It seems that most responses focus on performance. But is that in effect your greatest concern?

I also sense a request to understand the nature of the traffic. Is that correct?

[Philip_W]

Sorry, didn't have much time to follow up lately.

Yes, indeed Hugo - we were more concerned in detecting exactly what was causing the high CPU. We were thinking of taking a .pcap next time it happens.

[Timothy_Hall]

Try using cpview while the high CPU is happening and look at primarily CPU...Top Connections and secondarily Network...Top Connections.  Not sure how this info will be presented in the VSX environment but these screens can usually help identify elephant flows if that is indeed your issue, also take a look at the Connections/sec on the Overview screen in the event the CPU issue is caused by a burst of new connections and subsequent rule base lookup overhead.

[Philip_W]

When the issue reoccured yesterday, we took a .pcap and saw that the load balancers were keeping sending duplicate ACKs between each other. When the session was killed, the problem was solved.

Probably a bug in the loadbalancers...