CPU spiking on core 2 and core 6

[Blason_R]

Hi Team,

I have open server having 8 core processor and for last couple of days I am observing continuous CPU spike on Core 2 and core 6. This is R80.10. Surpringly my SND core is no spiking but only core 2, 6 & sometimes 7 is spiking.

This is carrying SSL VPN for around 450 users. We even increased rx-ringsize

Disabled URL filtering

Have put certain traffic on fast path using sim fastaccel

Any other help?

[Vladimir]

Check the traffic for the presence of the "elephant flows" to see if those are causing the issue: https://community.checkpoint.com/t5/General-Topics/R80-x-Performance-Tuning-Tip-Elephant-Flows-Heavy...

[the_rock]

I cant speak for anyone else, but the best way I found to solve issue like this is following (if its cluster, it has to be done on all members):

-cpconfig

-option for corexl, choose disable, reboot

-cpconfig again, re-enable corexl, reboot

-check again

Andy

[Timothy_Hall]

Almost certainly the presence of elephant flows as Vladimir said.  Use command fw ctl gconn to see what specific connections are being handled on the saturated cores, and fw ctl multik print_heavy_conn to see if the firewall has identified any elephant flows during the last 24 hours.  fastaccel can help if you correctly identify the elephant flows and fastpath them, but fastaccel will not work on corrections that must go F2F/slowpath for some reason.

[Blason_R]

Hi,

Thanks for the reply however considering R80.10 print heavy command gconn commands are not available. Is there any other way to identify?

[Timothy_Hall]

After the fact, no.  If you can catch the heavy utilization on those cores "in the act", try these live cpview screens (they won't be populated in cpview -t's historical mode):

CPU->Top Connections

Advanced->CoreXL->Instances-> FW-InstanceX->Top-FW-Lock consumers (you'll need to hit "a" to activate statistics)

If these screens don't help or are not available in your code version, you'll have to do it the old-fashioned way as described here: sk103293: How to get per CPU statistics and TOP FW-lock consumers with cpkstats

[Blason_R]

Yes I tried that but nothing specifically being pointed out. However I see connections destined to firewall IP with port 443 which are SSL VPN connections whenever CPUs are spiked and those are in F2F path.

[Vladimir]

I've tried to reply to you earlier, but it somehow did not post: If you have SmartEvent in your environment, you can try using "High Bandwidth Applications" view, filter it down to the timeframe of the incident and deep-dive in the results to pin-down the participants.

[Blason_R]

Hi Team,

Thanks for the heads-up. After struggling for about 2 weeks and really no response from TAC I finally upgraded the setup to 80.30 with maximum rx-ringsize and LACP created for LAN interface.

Seems like the no complaints received at least for past two days.