- Products
- Learn
- Local User Groups
- Partners
- More
Check Point Jump-Start Online Training
Now Available on CheckMates for Beginners!
Why do Hackers Love IoT Devices so Much?
Join our TechTalk on Aug 17, at 5PM CET | 11AM EST
Welcome to Maestro Masters!
Talk to Masters, Engage with Masters, Be a Maestro Master!
ZTNA Buyer’s Guide
Zero Trust essentials for your most valuable assets
The SMB Cyber Master
Boost your knowledge on Quantum Spark SMB gateways!
As YOU DESERVE THE BEST SECURITY
Upgrade to our latest GA Jumbo
CheckFlix!
All Videos In One Space
Hi Checkmates,
For the second week in a row, over the weekend we have been experiencing heavy (allowed) trafic through our VSX (R77.30) toward servers located behind our load balancers. This causes high CPU usage on 2 cores and now we are fearing some targetted DDOS or reconnaissance action is taking place.
We received no complaints from users or server admins. We know which VS is impacted but are having difficulties identifiying exactly what is happening.
To this end I used:
fw tab -u -t connections | awk '{ print $2 }' | sort -n | uniq -c | sort -nr | head -10
from 'My top 3 CLI commands' ( where Timothy shared a way of showing the top ten source IPs hogging slots in the connection table). This gave us some IPs, but basically we could see that in Smartlog too
How could we investigate this any deeper? Taking a .cap wouldn't help a lot, or would it?
Any ideas?
Hi Philip,
My first course of action would be to run Timothy's super seven commands as found here:
In addition, could you answer the following questions please:
1) What are the blades enabled on the affected VSX?
2) Are you aware of any changes that might have taken place two weeks ago?
3) What are the interesting services that are responsible for this increased CPU usage?
Thanks.
Hi Nick,
Thanks for pointing me toward Rick Hoppe's s7pac script, I'll run it asap.
1) Enabled blades on that VS: only fw & ips (i know...)
2) Any changes : the environment changes constantly. If anything changes (or if tests are done) this is not always communicated to the network/security guys so it is as good as impossible to find out.
3) Interesting services : we were seeing mainly http/https.
According to your description, you know the (DDOS) IP addresses.
I'd block the IP addresses. You can read this article
R80.x Performance Tuning Tip – DDoS „fw sam“ vs. „fwaccel dos“
On R77:30 ( is out of support in september 2019) use „fw sam“ to block DDOS IP‘s
On R80.10 and above block the DDOS IP on SecureXL layer.
# fwaccel dos blacklist -a <DDOS IP>
fw ctl zdebug + monitorall | grep -A 40 -B 20 <DDOS IP>
Now you can analyze the session in more detail.
Start Top and take a closer look at the process and cores.
top (+ buttom 1)
Check daemons with high CPU load. More see sk97638 Check Point Processes and Daemons
Check the affinity:
fw ctl affinity -l -a -v
Check enabled blades:
enabled_blades
Check the DDOS IP in the connection table:
fw tab -t connections -f -u | grep <DDOS IP>
Is "aggressive aging" enabled for this IP?
TIP:
Set stricter values in IPS for DDOS attacks.
Hi Guys,
Thanks for the great troubleshooting tips!
It seems that most responses focus on performance. But is that in effect your greatest concern?
I also sense a request to understand the nature of the traffic. Is that correct?
Sorry, didn't have much time to follow up lately.
Yes, indeed Hugo - we were more concerned in detecting exactly what was causing the high CPU. We were thinking of taking a .pcap next time it happens.
Try using cpview while the high CPU is happening and look at primarily CPU...Top Connections and secondarily Network...Top Connections. Not sure how this info will be presented in the VSX environment but these screens can usually help identify elephant flows if that is indeed your issue, also take a look at the Connections/sec on the Overview screen in the event the CPU issue is caused by a burst of new connections and subsequent rule base lookup overhead.
When the issue reoccured yesterday, we took a .pcap and saw that the load balancers were keeping sending duplicate ACKs between each other. When the session was killed, the problem was solved.
Probably a bug in the loadbalancers...
About CheckMates
Learn Check Point
Advanced Learning
YOU DESERVE THE BEST SECURITY