topic Re: high cpu allowed but unknown trafic in General Topics

high cpu allowed but unknown trafic

Philip_W — Mon, 08 Jul 2019 13:58:33 GMT

Hi Checkmates,

For the second week in a row, over the weekend we have been experiencing heavy (allowed) trafic through our VSX (R77.30) toward servers located behind our load balancers. This causes high CPU usage on 2 cores and now we are fearing some targetted DDOS or reconnaissance action is taking place.

We received no complaints from users or server admins. We know which VS is impacted but are having difficulties identifiying exactly what is happening.

To this end I used:

fw tab -u -t connections | awk '{ print $2 }' | sort -n | uniq -c | sort -nr | head -10

from 'My top 3 CLI commands' ( where Timothy shared a way of showing the top ten source IPs hogging slots in the connection table). This gave us some IPs, but basically we could see that in Smartlog too

How could we investigate this any deeper? Taking a .cap wouldn't help a lot, or would it?

Any ideas?

Re: high cpu allowed but unknown trafic

Nick_Doropoulos — Mon, 08 Jul 2019 18:17:45 GMT

Hi Philip,

My first course of action would be to run Timothy's super seven commands as found here:

https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac/td-p/40528

In addition, could you answer the following questions please:

1) What are the blades enabled on the affected VSX?

2) Are you aware of any changes that might have taken place two weeks ago?

3) What are the interesting services that are responsible for this increased CPU usage?

Thanks.

Re: high cpu allowed but unknown trafic

Philip_W — Tue, 09 Jul 2019 06:44:16 GMT

Hi Nick,

Thanks for pointing me toward Rick Hoppe's s7pac script, I'll run it asap.

1) Enabled blades on that VS: only fw & ips (i know...)

2) Any changes : the environment changes constantly. If anything changes (or if tests are done) this is not always communicated to the network/security guys so it is as good as impossible to find out.

3) Interesting services : we were seeing mainly http/https.

Re: high cpu allowed but unknown trafic

HeikoAnkenbrand — Tue, 09 Jul 2019 07:10:36 GMT

According to your description, you know the (DDOS) IP addresses.

I'd block the IP addresses. You can read this article

R80.x Performance Tuning Tip – DDoS „fw sam“ vs. „fwaccel dos“

On R77:30 ( is out of support in september 2019) use „fw sam“ to block DDOS IP‘s

On R80.10 and above block the DDOS IP on SecureXL layer.

# fwaccel dos blacklist -a <DDOS IP>

Re: high cpu allowed but unknown trafic

HeikoAnkenbrand — Tue, 09 Jul 2019 07:49:04 GMT

fw ctl zdebug + monitorall | grep -A 40 -B 20 <DDOS IP>

Now you can analyze the session in more detail.

Start Top and take a closer look at the process and cores.

top (+ buttom 1)

Check daemons with high CPU load. More see sk97638 Check Point Processes and Daemons

Check the affinity:

fw ctl affinity -l -a -v

Check enabled blades:

enabled_blades

Check the DDOS IP in the connection table:
fw tab -t connections -f -u | grep <DDOS IP>

Is "aggressive aging" enabled for this IP?

TIP:

Set stricter values in IPS for DDOS attacks.

Re: high cpu allowed but unknown trafic

Philip_W — Tue, 09 Jul 2019 07:30:04 GMT

Hi Guys,

Thanks for the great troubleshooting tips!

Re: high cpu allowed but unknown trafic

Hugo_vd_Kooij — Tue, 16 Jul 2019 08:08:14 GMT

It seems that most responses focus on performance. But is that in effect your greatest concern?

I also sense a request to understand the nature of the traffic. Is that correct?

Re: high cpu allowed but unknown trafic

Philip_W — Tue, 16 Jul 2019 12:56:57 GMT

Sorry, didn't have much time to follow up lately.

Yes, indeed Hugo - we were more concerned in detecting exactly what was causing the high CPU. We were thinking of taking a .pcap next time it happens.

Re: high cpu allowed but unknown trafic

Timothy_Hall — Tue, 16 Jul 2019 13:21:45 GMT

Try using cpview while the high CPU is happening and look at primarily CPU...Top Connections and secondarily Network...Top Connections. Not sure how this info will be presented in the VSX environment but these screens can usually help identify elephant flows if that is indeed your issue, also take a look at the Connections/sec on the Overview screen in the event the CPU issue is caused by a burst of new connections and subsequent rule base lookup overhead.

Re: high cpu allowed but unknown trafic

Philip_W — Fri, 19 Jul 2019 13:17:03 GMT

When the issue reoccured yesterday, we took a .pcap and saw that the load balancers were keeping sending duplicate ACKs between each other. When the session was killed, the problem was solved.

Probably a bug in the loadbalancers...