topic Re: CPU Spikes - help needed in General Topics

CPU Spikes - help needed

Teddy_Brewski — Wed, 10 Jun 2026 09:55:22 GMT

Hello,

A pair of new 9300 appliances running VSX R82 Take 91.

We're struggling with occasional 100% CPU spikes that last ~6-12 seconds and result in packet loss during the incident.
Out of 4 VSs, only two are affected and they cover publicly exposed authoritative DNS servers, so we suspect that we are under fast flood DNS attacks (we have a long history of fighting with those).

The following DDoS rules have been setup quite some time ago:

# fwaccel dos rate get fwaccel dos rate add -i "<xxx>" -action drop -log regular destination cidr:192.168.100.1 pkt-rate 600 service any fwaccel dos rate add -i "<xxx>" -action drop -log regular destination cidr:192.168.100.2 pkt-rate 600 service any fwaccel dos rate add -i "<xxx>" -action drop -log regular destination cidr:192.168.100.3 pkt-rate 600 service any (3 rules found)

I do see "Packets Dropped" increasing occasionally for all three rules, although couldn't pinpoint at exact time of the incident yet (it's happening out of office hours and last few seconds only).

Jun 9 17:54:42 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fwk4_0, start time: 09/06/26 17:54:30, spike duration (sec): 12, initial cpu usage: 87, average cpu usage: 73, perf taken: 1 Jun 9 17:58:36 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 3, top consumer: fwk4_0, start time: 09/06/26 17:58:30, spike duration (sec): 5, initial cpu usage: 89, average cpu usage: 89, perf taken: 0 Jun 9 18:00:02 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 8, top consumer: pkt_thread_6, start time: 09/06/26 17:59:55, spike duration (sec): 6, initial cpu usage: 94, average cpu usage: 94, perf taken: 1 Jun 9 18:00:53 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fwk4_0, start time: 09/06/26 18:00:47, spike duration (sec): 5, initial cpu usage: 100, average cpu usage: 100, perf taken: 0 Jun 9 18:01:33 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 3, top consumer: fwk4_0, start time: 09/06/26 18:01:21, spike duration (sec): 11, initial cpu usage: 94, average cpu usage: 97, perf taken: 0 Jun 9 18:01:33 2026 FWVSXN01 spike_detective: spike info: type: thread, thread id: 31933, thread name: fwk4_0, start time: 09/06/26 18:01:26, spike duration (sec): 6, initial cpu usage: 99, average cpu usage: 99, perf taken: 1 Jun 9 18:02:13 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 2, top consumer: fwk4_0, start time: 09/06/26 18:02:01, spike duration (sec): 11, initial cpu usage: 83, average cpu usage: 70, perf taken: 1 Jun 9 18:03:38 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 2, top consumer: fwk4_0, start time: 09/06/26 18:03:32, spike duration (sec): 6, initial cpu usage: 89, average cpu usage: 89, perf taken: 0 Jun 9 18:05:15 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 2, top consumer: fwk4_0, start time: 09/06/26 18:05:03, spike duration (sec): 12, initial cpu usage: 90, average cpu usage: 65, perf taken: 1 Jun 9 18:05:15 2026 FWVSXN01 spike_detective: spike info: type: thread, thread id: 31933, thread name: fwk4_0, start time: 09/06/26 18:05:09, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 1 Jun 9 23:06:20 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 10, top consumer: fwk5_0, start time: 09/06/26 23:06:14, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 1 Jun 9 23:06:32 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 12, top consumer: fwk5_0, start time: 09/06/26 23:06:25, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 0 Jun 9 23:06:49 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 14, top consumer: fwk5_0, start time: 09/06/26 23:06:42, spike duration (sec): 6, initial cpu usage: 100, average cpu usage: 100, perf taken: 0 Jun 9 23:07:40 2026 FWVSXN01 spike_detective: spike info: type: thread, thread id: 17787, thread name: fwk5_0, start time: 09/06/26 23:07:34, spike duration (sec): 5, initial cpu usage: 99, average cpu usage: 99, perf taken: 1 Jun 9 23:07:46 2026 FWVSXN01 spike_detective: spike info: type: cpu, cpu core: 10, top consumer: fwk5_0, start time: 09/06/26 23:07:28, spike duration (sec): 17, initial cpu usage: 99, average cpu usage: 93, perf taken: 0

Checking cpview shows high CPU with no noticeable increase in Concurrent Connections.

Any tips/hints/ideas would be greatly appreciated!

Thank you.

Re: CPU Spikes - help needed

Timothy_Hall — Wed, 10 Jun 2026 12:22:48 GMT

Probably an occasional short-lived elephant flow traversing the Medium or Slow paths at LAN speeds. What does fw ctl multik print_heavy_conn show as far as elephant flows for the last 24 hours? The spike detective should run this command when a spike detection is made (along with top_conns) and log it to /var/log/spike_detective/data_spike_thread_*. These models also have P and E cores, and if an elephant flow hits an E-core, it is much easier for that core to get spiked than a P-core.

Re: CPU Spikes - help needed

Teddy_Brewski — Wed, 10 Jun 2026 12:47:17 GMT

Thanks for your reply @Timothy_Hall

[fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 61%; Connection instance load: 86%; StartTime: 09/06/26 23:03:38; Duration: 94; IdentificationTime: 09/06/26 23:05:17; Service: 6:80; Total Packets: 1673; Total Bytes: 99192; [fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 70%; Connection instance load: 96%; StartTime: 09/06/26 23:03:38; Duration: 13; IdentificationTime: 09/06/26 23:07:26; Service: 6:80; Total Packets: 136; Total Bytes: 7501; [fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 77%; Connection instance load: 95%; StartTime: 09/06/26 23:03:39; Duration: 17; IdentificationTime: 09/06/26 23:08:34; Service: 6:80; Total Packets: 121; Total Bytes: 6860; [fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 78%; Connection instance load: 92%; StartTime: 09/06/26 23:03:38; Duration: 14; IdentificationTime: 09/06/26 23:07:01; Service: 6:80; Total Packets: 148; Total Bytes: 8254; [fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 81%; Connection instance load: 91%; StartTime: 09/06/26 23:03:38; Duration: 15; IdentificationTime: 09/06/26 23:08:09; Service: 6:80; Total Packets: 126; Total Bytes: 6996; [fw_0]; Conn: 177.101.43.163:40677 -> 192.168.0.254:80 IPP 6; Instance load: 66%; Connection instance load: 95%; StartTime: 09/06/26 23:03:38; Duration: 12; IdentificationTime: 09/06/26 23:07:49; Service: 6:80; Total Packets: 103; Total Bytes: 5670;

Where 192.168.0.254 is the IP assigned to the DMZ interface of the firewall.

And from the other VS:

[fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 60%; Connection instance load: 98%; StartTime: 09/06/26 17:50:15; Duration: 33; IdentificationTime: 09/06/26 18:04:46; Service: 6:80; Total Packets: 100; Total Bytes: 5528; [fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 65%; Connection instance load: 99%; StartTime: 09/06/26 17:50:15; Duration: 183; IdentificationTime: 09/06/26 17:52:40; Service: 6:80; Total Packets: 1230; Total Bytes: 67600; [fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 61%; Connection instance load: 99%; StartTime: 09/06/26 17:50:16; Duration: 132; IdentificationTime: 09/06/26 18:00:13; Service: 6:80; Total Packets: 418; Total Bytes: 23003; [fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 73%; Connection instance load: 99%; StartTime: 09/06/26 17:50:15; Duration: 83; IdentificationTime: 09/06/26 17:50:47; Service: 6:80; Total Packets: 1994; Total Bytes: 110540; [fw_0]; Conn: 78.120.76.69:58714 -> 192.168.0.30:80 IPP 6; Instance load: 60%; Connection instance load: 99%; StartTime: 09/06/26 17:50:15; Duration: 148; IdentificationTime: 09/06/26 17:56:10; Service: 6:80; Total Packets: 585; Total Bytes: 32245;

Re: CPU Spikes - help needed

CaseyB — Wed, 10 Jun 2026 12:58:17 GMT

Upgrade to JHF-103.

I had multiple CPU spike issues on 91 and two production issues. After talking with TAC we decided to upgrade, and I have not seen the same symptoms.

Re: CPU Spikes - help needed

Lesley — Wed, 10 Jun 2026 13:03:55 GMT

If you have a DNS server that can be reached from outside it acts like a big honey pot. I think it this cause you are affected by short DDOS attacks that are to short for you to notice in cpview (cpview prints out every 1 minute). Maybe the traffic logs have any hints of there is indeed an attack during the outage you have.

To help protect, check if the relevant IPS protections are enabled. Please note inspection costs CPU and could maybe be the cause of the issue.

Check also this SK:

https://support.checkpoint.com/results/sk/sk112241

3. Allocate / configure sufficient Internet bandwidth to Security Gateway.

8. Activate and configure IPS 'Geo Protection' protection / Geo Policy.

13. Block traffic coming from known malicious IP addresses

14. Enable and configure SecureXL Penalty Box <--- This one only works for dropped traffic! (not allowed by rulebase)

Re: CPU Spikes - help needed

Teddy_Brewski — Wed, 10 Jun 2026 13:17:40 GMT

Thank you for the links @Lesley. As per 'fw ctl multik print_heavy_conn' it doesn't seem to be the case this time.

Re: CPU Spikes - help needed

Timothy_Hall — Wed, 10 Jun 2026 13:22:04 GMT

Those remote IP addresses appear to be in Brazil and France. Do you have operations/customers/partners/vendors in those countries? Due to the short duration, it could also be the following, although these remote networks do not appear to be part of Check Point:

sk183868: Daily short-lived 100% CPU spikes on Quantum Security Gateways
sk174347: Software blade updates may cause single CPU spike

Re: CPU Spikes - help needed

Lesley — Wed, 10 Jun 2026 13:32:51 GMT

True could still be bug, would update indeed. And after that follow tips above. They improve security anyway, if it does not solve this issue you still have a good argument to have spent time on it.

Re: CPU Spikes - help needed

Teddy_Brewski — Thu, 30 Jul 2026 12:24:55 GMT

Thank you all, unfortunately none of this helped. We've experienced an intermittent connectivity again (with 80% packet loss and 70-90% CPU load) for some 20 minutes.

This looks like some sort of DDoS:

[fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 69%; Connection instance load: 90%; StartTime: 30/07/26 11:00:36; Duration: 15; IdentificationTime: 30/07/26 11:26:39; Service: 6:80; Total Packets: 21; Total Bytes: 1308; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 71%; StartTime: 30/07/26 11:00:36; Duration: 204; IdentificationTime: 30/07/26 11:14:49; Service: 6:80; Total Packets: 909; Total Bytes: 50141; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 65%; Connection instance load: 92%; StartTime: 30/07/26 11:00:36; Duration: 2; IdentificationTime: 30/07/26 11:30:50; Service: 6:80; Total Packets: 7; Total Bytes: 391; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 90%; StartTime: 30/07/26 11:00:36; Duration: 42; IdentificationTime: 30/07/26 11:29:46; Service: 6:80; Total Packets: 60; Total Bytes: 3748; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 62%; Connection instance load: 94%; StartTime: 30/07/26 11:00:36; Duration: 7; IdentificationTime: 30/07/26 11:28:28; Service: 6:80; Total Packets: 10; Total Bytes: 574; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 64%; Connection instance load: 91%; StartTime: 30/07/26 11:00:37; Duration: 6; IdentificationTime: 30/07/26 11:29:13; Service: 6:80; Total Packets: 13; Total Bytes: 737; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 66%; Connection instance load: 93%; StartTime: 30/07/26 11:00:36; Duration: 3; IdentificationTime: 30/07/26 11:31:18; Service: 6:80; Total Packets: 2; Total Bytes: 114; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 80%; StartTime: 30/07/26 11:00:36; Duration: 16; IdentificationTime: 30/07/26 11:27:54; Service: 6:80; Total Packets: 24; Total Bytes: 1415; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 72%; StartTime: 30/07/26 11:00:36; Duration: 8; IdentificationTime: 30/07/26 11:30:33; Service: 6:80; Total Packets: 10; Total Bytes: 549; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 66%; Connection instance load: 95%; StartTime: 30/07/26 11:00:36; Duration: 3; IdentificationTime: 30/07/26 11:32:46; Service: 6:80; Total Packets: 2; Total Bytes: 108; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 69%; Connection instance load: 96%; StartTime: 30/07/26 11:00:36; Duration: 13; IdentificationTime: 30/07/26 11:28:56; Service: 6:80; Total Packets: 25; Total Bytes: 1583; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 61%; Connection instance load: 91%; StartTime: 30/07/26 11:00:36; Duration: 7; IdentificationTime: 30/07/26 11:31:02; Service: 6:80; Total Packets: 8; Total Bytes: 440; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 88%; StartTime: 30/07/26 11:00:36; Duration: 6; IdentificationTime: 30/07/26 11:32:31; Service: 6:80; Total Packets: 6; Total Bytes: 331; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 61%; Connection instance load: 65%; StartTime: 30/07/26 11:00:37; Duration: 73; IdentificationTime: 30/07/26 11:13:30; Service: 6:80; Total Packets: 265; Total Bytes: 14700; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 61%; Connection instance load: 95%; StartTime: 30/07/26 11:00:36; Duration: 14; IdentificationTime: 30/07/26 11:31:30; Service: 6:80; Total Packets: 18; Total Bytes: 1121; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 93%; StartTime: 30/07/26 11:00:36; Duration: 3; IdentificationTime: 30/07/26 11:32:22; Service: 6:80; Total Packets: 2; Total Bytes: 114; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 79%; StartTime: 30/07/26 11:00:37; Duration: 4; IdentificationTime: 30/07/26 11:27:03; Service: 6:80; Total Packets: 8; Total Bytes: 448; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 69%; Connection instance load: 90%; StartTime: 30/07/26 11:00:36; Duration: 4; IdentificationTime: 30/07/26 11:28:43; Service: 6:80; Total Packets: 7; Total Bytes: 392; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 83%; StartTime: 30/07/26 11:00:37; Duration: 19; IdentificationTime: 30/07/26 11:26:16; Service: 6:80; Total Packets: 35; Total Bytes: 2083; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 68%; Connection instance load: 88%; StartTime: 30/07/26 11:00:37; Duration: 26; IdentificationTime: 30/07/26 11:27:15; Service: 6:80; Total Packets: 49; Total Bytes: 2935; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 63%; Connection instance load: 91%; StartTime: 30/07/26 11:00:37; Duration: 8; IdentificationTime: 30/07/26 11:29:36; Service: 6:80; Total Packets: 15; Total Bytes: 873; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 66%; Connection instance load: 94%; StartTime: 30/07/26 11:00:36; Duration: 12; IdentificationTime: 30/07/26 11:32:02; Service: 6:80; Total Packets: 12; Total Bytes: 821; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 60%; Connection instance load: 86%; StartTime: 30/07/26 11:00:36; Duration: 422; IdentificationTime: 30/07/26 11:19:07; Service: 6:80; Total Packets: 1357; Total Bytes: 77355; [fw_0]; Conn: 143.105.136.60:63580 -> XXX.YYY.ZZZ:80 IPP 6; Instance load: 65%; Connection instance load: 85%; StartTime: 30/07/26 11:00:37; Duration: 9; IdentificationTime: 30/07/26 11:29:24; Service: 6:80; Total Packets: 11; Total Bytes: 650;

What I noticed so far is that it seems that it only happens when they target the public IPs of the cluster itself. In all previous cases, including the recent one, XXX.YYY.ZZZ is the firewall cluster IP (DMZ in this case).

I also noticed that it's always port 80, so I disabled it with 'fw ctl set int multi_portal_allow_redirect 0' since this cluster is used for remote access.

Any other hints? How's it possible that a single IP from the other side of the world (we're in Europe and 143.105.136.60 seems to be in Argentina) could cause such a catastrophic impact? Our 9300 cluster was on its knees.

Re: CPU Spikes - help needed

PhoneBoy — Thu, 30 Jul 2026 14:49:08 GMT

I'd engage TAC with this if you haven't already.

Re: CPU Spikes - help needed

Lesley — Thu, 30 Jul 2026 20:30:15 GMT

1 host in theory could cause high impact on firewall performance. Maybe it was a bunch of machine hidden behind one public IP. The firewall, in this case port 80 will do loads of inspection because traffic is not encrypted. How the traffic logs look like? Any blades kicking in from this IP? What blades are enabled? Are you using syn defender?

Re: CPU Spikes - help needed

Matthias42 — Fri, 31 Jul 2026 13:23:38 GMT

Not sure, if this fits your problem, but I had similar problems in the past. For me, they go away if i "drop without log" instead of "drop+log". Logging seems to be a real resource hog. Some gateways seem to block themself, if they cannot send logs fast enough to the log server, to avoid losing logs.

Re: CPU Spikes - help needed

Teddy_Brewski — Fri, 31 Jul 2026 16:56:21 GMT

Don't see anything unusual. The host was scanning multiple networks via multiple ports until it hit the VIP of the DMZ interface. The implied rule triggered allowing access over 80/tcp and after that things went bad.

cpview showed regular values for the number of concurrent connections and there was nothing unusual with the load of the Internet line. CPU jumped to 100% resulting in packet loss.

The only blade that is enabled is Firewall. The rule that triggered was implied.

Syn defender is not activated:

# fwaccel synatk config Loaded from Policy enabled 0 enforce 1 global_high_threshold 10000 high_threshold 5000 low_threshold 1000 periodic_updates 1 cookie_resolution_shift 6 min_frag_sz 80 cookie_mss_v4 0 cookie_window_scale -1 cookie_sack_permitted 0 score_alpha 100 monitor_log_interval (msec) 60000 grace_timeout (msec) 30000 min_time_in_active (msec) 60000

Things went back to normal when I blocked the IP range on the upstream router.