topic Re: CPU spiking on core 2 and core 6 in General Topics

CPU spiking on core 2 and core 6

Blason_R — Mon, 01 Mar 2021 17:17:20 GMT

Hi Team,

I have open server having 8 core processor and for last couple of days I am observing continuous CPU spike on Core 2 and core 6. This is R80.10. Surpringly my SND core is no spiking but only core 2, 6 & sometimes 7 is spiking.

This is carrying SSL VPN for around 450 users. We even increased rx-ringsize

Disabled URL filtering

Have put certain traffic on fast path using sim fastaccel

Any other help?

Re: CPU spiking on core 2 and core 6

Vladimir — Mon, 01 Mar 2021 18:01:20 GMT

Check the traffic for the presence of the "elephant flows" to see if those are causing the issue: https://community.checkpoint.com/t5/General-Topics/R80-x-Performance-Tuning-Tip-Elephant-Flows-Heavy-Connections/m-p/69105

Re: CPU spiking on core 2 and core 6

the_rock — Mon, 01 Mar 2021 18:47:32 GMT

I cant speak for anyone else, but the best way I found to solve issue like this is following (if its cluster, it has to be done on all members):

-cpconfig

-option for corexl, choose disable, reboot

-cpconfig again, re-enable corexl, reboot

-check again

Andy

Re: CPU spiking on core 2 and core 6

Timothy_Hall — Mon, 01 Mar 2021 19:33:01 GMT

Almost certainly the presence of elephant flows as Vladimir said. Use command fw ctl gconn to see what specific connections are being handled on the saturated cores, and fw ctl multik print_heavy_conn to see if the firewall has identified any elephant flows during the last 24 hours. fastaccel can help if you correctly identify the elephant flows and fastpath them, but fastaccel will not work on corrections that must go F2F/slowpath for some reason.

Re: CPU spiking on core 2 and core 6

Blason_R — Tue, 02 Mar 2021 00:25:06 GMT

Hi,

Thanks for the reply however considering R80.10 print heavy command gconn commands are not available. Is there any other way to identify?

Re: CPU spiking on core 2 and core 6

Timothy_Hall — Tue, 02 Mar 2021 04:17:16 GMT

After the fact, no. If you can catch the heavy utilization on those cores "in the act", try these live cpview screens (they won't be populated in cpview -t's historical mode):

CPU->Top Connections

Advanced->CoreXL->Instances-> FW-InstanceX->Top-FW-Lock consumers (you'll need to hit "a" to activate statistics)

If these screens don't help or are not available in your code version, you'll have to do it the old-fashioned way as described here: sk103293: How to get per CPU statistics and TOP FW-lock consumers with cpkstats

Re: CPU spiking on core 2 and core 6

Blason_R — Wed, 03 Mar 2021 13:52:40 GMT

Yes I tried that but nothing specifically being pointed out. However I see connections destined to firewall IP with port 443 which are SSL VPN connections whenever CPUs are spiked and those are in F2F path.

Re: CPU spiking on core 2 and core 6

Vladimir — Wed, 03 Mar 2021 14:09:49 GMT

I've tried to reply to you earlier, but it somehow did not post: If you have SmartEvent in your environment, you can try using "High Bandwidth Applications" view, filter it down to the timeframe of the incident and deep-dive in the results to pin-down the participants.

Re: CPU spiking on core 2 and core 6

Blason_R — Tue, 09 Mar 2021 06:34:30 GMT

Hi Team,

Thanks for the heads-up. After struggling for about 2 weeks and really no response from TAC I finally upgraded the setup to 80.30 with maximum rx-ringsize and LACP created for LAN interface.

Seems like the no complaints received at least for past two days.