CPU Spikes - help needed

[Teddy_Brewski]

Hello,

A pair of new 9300 appliances running VSX R82 Take 91.

We're struggling with occasional 100% CPU spikes that last ~6-12 seconds and result in packet loss during the incident.
Out of 4 VSs, only two are affected and they cover publicly exposed authoritative DNS servers, so we suspect that we are under fast flood DNS attacks (we have a long history of fighting with those).

The following DDoS rules have been setup quite some time ago:

# fwaccel dos rate get
fwaccel dos rate add -i "<xxx>" -action drop -log regular destination cidr:192.168.100.1 pkt-rate 600 service any 
fwaccel dos rate add -i "<xxx>" -action drop -log regular destination cidr:192.168.100.2 pkt-rate 600 service any 
fwaccel dos rate add -i "<xxx>" -action drop -log regular destination cidr:192.168.100.3 pkt-rate 600 service any 
(3 rules found)

I do see "Packets Dropped" increasing occasionally for all three rules, although couldn't pinpoint at exact time of the incident yet (it's happening out of office hours and last few seconds only).

Jun  9 17:54:42 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fwk4_0, start time: 09/06/26 17:54:30, spike duration (sec): 12, initial cpu usage: 87, average cpu usage: 73, perf taken: 1
Jun  9 17:58:36 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 3, top consumer: fwk4_0, start time: 09/06/26 17:58:30, spike duration (sec): 5, initial cpu usage: 89, average cpu usage: 89, perf taken: 0
Jun  9 18:00:02 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 8, top consumer: pkt_thread_6, start time: 09/06/26 17:59:55, spike duration (sec): 6, initial cpu usage: 94, average cpu usage: 94, perf taken: 1
Jun  9 18:00:53 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fwk4_0, start time: 09/06/26 18:00:47, spike duration (sec): 5, initial cpu usage: 100, average cpu usage: 100, perf taken: 0
Jun  9 18:01:33 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 3, top consumer: fwk4_0, start time: 09/06/26 18:01:21, spike duration (sec): 11, initial cpu usage: 94, average cpu usage: 97, perf taken: 0
Jun  9 18:01:33 2026 FWVSXN01 spike_detective: spike info: type: thread, thread id: 31933, thread name: fwk4_0, start time: 09/06/26 18:01:26, spike duration (sec): 6, initial cpu usage: 99, average cpu usage: 99, perf taken: 1
Jun  9 18:02:13 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 2, top consumer: fwk4_0, start time: 09/06/26 18:02:01, spike duration (sec): 11, initial cpu usage: 83, average cpu usage: 70, perf taken: 1
Jun  9 18:03:38 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 2, top consumer: fwk4_0, start time: 09/06/26 18:03:32, spike duration (sec): 6, initial cpu usage: 89, average cpu usage: 89, perf taken: 0
Jun  9 18:05:15 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 2, top consumer: fwk4_0, start time: 09/06/26 18:05:03, spike duration (sec): 12, initial cpu usage: 90, average cpu usage: 65, perf taken: 1
Jun  9 18:05:15 2026 FWVSXN01 spike_detective: spike info: type: thread, thread id: 31933, thread name: fwk4_0, start time: 09/06/26 18:05:09, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 1

Jun  9 23:06:20 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 10, top consumer: fwk5_0, start time: 09/06/26 23:06:14, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 1
Jun  9 23:06:32 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fwk5_0, start time: 09/06/26 23:06:25, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 0
Jun  9 23:06:49 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 14, top consumer: fwk5_0, start time: 09/06/26 23:06:42, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 0
Jun  9 23:07:40 2026 FWVSXN01 spike_detective: spike info: type: thread, thread id: 17787, thread name: fwk5_0, start time: 09/06/26 23:07:34, spike duration (sec): 5, initial cpu usage: 99, average cpu usage: 99, perf taken: 1
Jun  9 23:07:46 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 10, top consumer: fwk5_0, start time: 09/06/26 23:07:28, spike duration (sec): 17, initial cpu usage: 99, average cpu usage: 93, perf taken: 0

Checking cpview shows high CPU with no noticeable increase in Concurrent Connections.

Any tips/hints/ideas would be greatly appreciated!

Thank you.

[Timothy_Hall]

Probably an occasional short-lived elephant flow traversing the Medium or Slow paths at LAN speeds.  What does fw ctl multik print_heavy_conn show as far as elephant flows for the last 24 hours?  The spike detective should run this command when a spike detection is made (along with top_conns) and log it to /var/log/spike_detective/data_spike_thread_*.  These models also have P and E cores, and if an elephant flow hits an E-core, it is much easier for that core to get spiked than a P-core.

[Teddy_Brewski]

Thanks for your reply @Timothy_Hall 

[fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 61%; Connection instance load: 86%; StartTime: 09/06/26 23:03:38; Duration: 94; IdentificationTime: 09/06/26 23:05:17; Service: 6:80; Total Packets: 1673; Total Bytes: 99192; 
[fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 70%; Connection instance load: 96%; StartTime: 09/06/26 23:03:38; Duration: 13; IdentificationTime: 09/06/26 23:07:26; Service: 6:80; Total Packets: 136; Total Bytes: 7501; 
[fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 77%; Connection instance load: 95%; StartTime: 09/06/26 23:03:39; Duration: 17; IdentificationTime: 09/06/26 23:08:34; Service: 6:80; Total Packets: 121; Total Bytes: 6860; 
[fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 78%; Connection instance load: 92%; StartTime: 09/06/26 23:03:38; Duration: 14; IdentificationTime: 09/06/26 23:07:01; Service: 6:80; Total Packets: 148; Total Bytes: 8254; 
[fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 81%; Connection instance load: 91%; StartTime: 09/06/26 23:03:38; Duration: 15; IdentificationTime: 09/06/26 23:08:09; Service: 6:80; Total Packets: 126; Total Bytes: 6996; 
[fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 66%; Connection instance load: 95%; StartTime: 09/06/26 23:03:38; Duration: 12; IdentificationTime: 09/06/26 23:07:49; Service: 6:80; Total Packets: 103; Total Bytes: 5670; 

Where 192.168.0.254 is the IP assigned to the DMZ interface of the firewall.

And from the other VS:

[fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 60%; Connection instance load: 98%; StartTime: 09/06/26 17:50:15; Duration: 33; IdentificationTime: 09/06/26 18:04:46; Service: 6:80; Total Packets: 100; Total Bytes: 5528; 
[fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 65%; Connection instance load: 99%; StartTime: 09/06/26 17:50:15; Duration: 183; IdentificationTime: 09/06/26 17:52:40; Service: 6:80; Total Packets: 1230; Total Bytes: 67600; 
[fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 61%; Connection instance load: 99%; StartTime: 09/06/26 17:50:16; Duration: 132; IdentificationTime: 09/06/26 18:00:13; Service: 6:80; Total Packets: 418; Total Bytes: 23003; 
[fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 73%; Connection instance load: 99%; StartTime: 09/06/26 17:50:15; Duration: 83; IdentificationTime: 09/06/26 17:50:47; Service: 6:80; Total Packets: 1994; Total Bytes: 110540; 
[fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 60%; Connection instance load: 99%; StartTime: 09/06/26 17:50:15; Duration: 148; IdentificationTime: 09/06/26 17:56:10; Service: 6:80; Total Packets: 585; Total Bytes: 32245;

[Lesley]

If you have a DNS server that can be reached from outside it acts like a big honey pot. I think it this cause you are affected by short DDOS attacks that are to short for you to notice in cpview (cpview prints out every 1 minute). Maybe the traffic logs have any hints of there is indeed an attack during the outage you have. 

To help protect, check if the relevant IPS protections are enabled. Please note inspection costs CPU and could maybe be the cause of the issue.

Check also this SK:

https://support.checkpoint.com/results/sk/sk112241

3. Allocate / configure sufficient Internet bandwidth to Security Gateway.

8. Activate and configure IPS 'Geo Protection' protection / Geo Policy.

13. Block traffic coming from known malicious IP addresses

14. Enable and configure SecureXL Penalty Box <--- This one only works for dropped traffic! (not allowed by rulebase)

[Teddy_Brewski]

Thank you for the links @Lesley. As per 'fw ctl multik print_heavy_conn' it doesn't seem to be the case this time.

[Timothy_Hall]

Those remote IP addresses appear to be in Brazil and France. Do you have operations/customers/partners/vendors in those countries?  Due to the short duration, it could also be the following, although these remote networks do not appear to be part of Check Point: 

• sk183868: Daily short-lived 100% CPU spikes on Quantum Security Gateways
• sk174347: Software blade updates may cause single CPU spike

[CaseyB]

Upgrade to JHF-103.

I had multiple CPU spike issues on 91 and two production issues. After talking with TAC we decided to upgrade, and I have not seen the same symptoms.

[Lesley]

True could still be bug, would update indeed. And after that follow tips above. They improve security anyway, if it does not solve this issue you still have a good argument to have spent time on it. 

[Teddy_Brewski]

Thank you all, unfortunately none of this helped. We've experienced an intermittent connectivity again (with 80% packet loss and 70-90% CPU load) for some 20 minutes.

This looks like some sort of DDoS:

[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 69%; Connection instance load: 90%; StartTime: 30/07/26 11:00:36; Duration: 15; IdentificationTime: 30/07/26 11:26:39; Service: 6:80; Total Packets: 21; Total Bytes: 1308; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 71%; StartTime: 30/07/26 11:00:36; Duration: 204; IdentificationTime: 30/07/26 11:14:49; Service: 6:80; Total Packets: 909; Total Bytes: 50141; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 65%; Connection instance load: 92%; StartTime: 30/07/26 11:00:36; Duration: 2; IdentificationTime: 30/07/26 11:30:50; Service: 6:80; Total Packets: 7; Total Bytes: 391; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 90%; StartTime: 30/07/26 11:00:36; Duration: 42; IdentificationTime: 30/07/26 11:29:46; Service: 6:80; Total Packets: 60; Total Bytes: 3748; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 62%; Connection instance load: 94%; StartTime: 30/07/26 11:00:36; Duration: 7; IdentificationTime: 30/07/26 11:28:28; Service: 6:80; Total Packets: 10; Total Bytes: 574; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 64%; Connection instance load: 91%; StartTime: 30/07/26 11:00:37; Duration: 6; IdentificationTime: 30/07/26 11:29:13; Service: 6:80; Total Packets: 13; Total Bytes: 737; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 66%; Connection instance load: 93%; StartTime: 30/07/26 11:00:36; Duration: 3; IdentificationTime: 30/07/26 11:31:18; Service: 6:80; Total Packets: 2; Total Bytes: 114; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 80%; StartTime: 30/07/26 11:00:36; Duration: 16; IdentificationTime: 30/07/26 11:27:54; Service: 6:80; Total Packets: 24; Total Bytes: 1415; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 72%; StartTime: 30/07/26 11:00:36; Duration: 8; IdentificationTime: 30/07/26 11:30:33; Service: 6:80; Total Packets: 10; Total Bytes: 549; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 66%; Connection instance load: 95%; StartTime: 30/07/26 11:00:36; Duration: 3; IdentificationTime: 30/07/26 11:32:46; Service: 6:80; Total Packets: 2; Total Bytes: 108; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 69%; Connection instance load: 96%; StartTime: 30/07/26 11:00:36; Duration: 13; IdentificationTime: 30/07/26 11:28:56; Service: 6:80; Total Packets: 25; Total Bytes: 1583; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 61%; Connection instance load: 91%; StartTime: 30/07/26 11:00:36; Duration: 7; IdentificationTime: 30/07/26 11:31:02; Service: 6:80; Total Packets: 8; Total Bytes: 440; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 88%; StartTime: 30/07/26 11:00:36; Duration: 6; IdentificationTime: 30/07/26 11:32:31; Service: 6:80; Total Packets: 6; Total Bytes: 331; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 61%; Connection instance load: 65%; StartTime: 30/07/26 11:00:37; Duration: 73; IdentificationTime: 30/07/26 11:13:30; Service: 6:80; Total Packets: 265; Total Bytes: 14700; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 61%; Connection instance load: 95%; StartTime: 30/07/26 11:00:36; Duration: 14; IdentificationTime: 30/07/26 11:31:30; Service: 6:80; Total Packets: 18; Total Bytes: 1121; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 93%; StartTime: 30/07/26 11:00:36; Duration: 3; IdentificationTime: 30/07/26 11:32:22; Service: 6:80; Total Packets: 2; Total Bytes: 114; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 79%; StartTime: 30/07/26 11:00:37; Duration: 4; IdentificationTime: 30/07/26 11:27:03; Service: 6:80; Total Packets: 8; Total Bytes: 448; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 69%; Connection instance load: 90%; StartTime: 30/07/26 11:00:36; Duration: 4; IdentificationTime: 30/07/26 11:28:43; Service: 6:80; Total Packets: 7; Total Bytes: 392; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 83%; StartTime: 30/07/26 11:00:37; Duration: 19; IdentificationTime: 30/07/26 11:26:16; Service: 6:80; Total Packets: 35; Total Bytes: 2083; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 88%; StartTime: 30/07/26 11:00:37; Duration: 26; IdentificationTime: 30/07/26 11:27:15; Service: 6:80; Total Packets: 49; Total Bytes: 2935; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 91%; StartTime: 30/07/26 11:00:37; Duration: 8; IdentificationTime: 30/07/26 11:29:36; Service: 6:80; Total Packets: 15; Total Bytes: 873; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 66%; Connection instance load: 94%; StartTime: 30/07/26 11:00:36; Duration: 12; IdentificationTime: 30/07/26 11:32:02; Service: 6:80; Total Packets: 12; Total Bytes: 821; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 60%; Connection instance load: 86%; StartTime: 30/07/26 11:00:36; Duration: 422; IdentificationTime: 30/07/26 11:19:07; Service: 6:80; Total Packets: 1357; Total Bytes: 77355; 
[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 65%; Connection instance load: 85%; StartTime: 30/07/26 11:00:37; Duration: 9; IdentificationTime: 30/07/26 11:29:24; Service: 6:80; Total Packets: 11; Total Bytes: 650;

What I noticed so far is that it seems that it only happens when they target the public IPs of the cluster itself. In all previous cases, including the recent one, XXX.YYY.ZZZ is the firewall cluster IP (DMZ in this case).

I also noticed that it's always port 80, so I disabled it with 'fw ctl set int multi_portal_allow_redirect 0' since this cluster is used for remote access.

Any other hints? How's it possible that a single IP from the other side of the world (we're in Europe and 143.105.136.60 seems to be in Argentina) could cause such a catastrophic impact? Our 9300 cluster was on its knees.

[PhoneBoy]

I'd engage TAC with this if you haven't already.

[Lesley]

1 host in theory could cause high impact on firewall performance. Maybe it was a bunch of machine hidden behind one public IP. The firewall, in this case port 80 will do loads of inspection because traffic is not encrypted. How the traffic logs look like? Any blades kicking in from this IP? What blades are enabled? Are you using syn defender? 

[Teddy_Brewski]

Don't see anything unusual. The host was scanning multiple networks via multiple ports until it hit the VIP of the DMZ interface. The implied rule triggered allowing access over 80/tcp and after that things went bad.

cpview showed regular values for the number of concurrent connections and there was nothing unusual with the load of the Internet line. CPU jumped to 100% resulting in packet loss.

The only blade that is enabled is Firewall. The rule that triggered was implied.

Syn defender is not activated:

# fwaccel synatk config

Loaded from                        Policy
enabled                                 0
enforce                                 1
global_high_threshold               10000
high_threshold                       5000
low_threshold                        1000
periodic_updates                        1
cookie_resolution_shift                 6
min_frag_sz                            80
cookie_mss_v4                           0
cookie_window_scale                    -1
cookie_sack_permitted                   0
score_alpha                           100
monitor_log_interval (msec)         60000
grace_timeout (msec)                30000
min_time_in_active (msec)           60000

Things went back to normal when I blocked the IP range on the upstream router.

[Matthias42]

Not sure, if this fits your problem, but I had similar problems in the past. For me, they go away if i "drop without log" instead of "drop+log". Logging seems to be a real resource hog. Some gateways seem to block themself, if they cannot send logs fast enough to the log server, to avoid losing logs.