topic Re: High Cpu utilization - Internet Speed drops to very low level in Firewall and Security Management

High Cpu utilization - Internet Speed drops to very low level

Netadmin2020 — Tue, 22 Dec 2020 05:27:15 GMT

Dear Colleagues ,

I have a problem with high CPU utilization and by the time of this issue the internet speed to go down dramatically.

Appliance Check Point 15600 - Hot fix: Check_Point_R80_30_JUMBO_HF_Bundle_T219_sk153152

I am attaching some photos from the commands executed by the time of issue.

FYI we are using Policy Based Routing & Remote Access VPN.

Re: High Cpu utilization - Internet Speed drops to very low level

Magnus-Holmberg — Tue, 22 Dec 2020 06:55:02 GMT

Are you using multi queue on the interface? fw ctl affinity –l
could you show enabled_blades as well?

any messages in var/log/messages?

Re: High Cpu utilization - Internet Speed drops to very low level

Netadmin2020 — Tue, 22 Dec 2020 10:05:53 GMT

Re: High Cpu utilization - Internet Speed drops to very low level

Magnus-Holmberg — Tue, 22 Dec 2020 10:36:43 GMT

Where is your traffic going in and out?

CPU0 and that is allocated to Eth3-02, Eth2-06
Based on your picture above, this CPU core goes full.
What are those interfaces used for, if its used for traffic i would activate multi Q so that you can dist the load over multiple cores.
https://sc1.checkpoint.com/documents/R80.30/WebAdminGuides/EN/CP_R80.30_PerformanceTuning_AdminGuide/html_frameset.htm?topic=documents/R80.30/WebAdminGuides/EN/CP_R80.30_PerformanceTuning_AdminGuide/206538

Regards,
Magnus

Re: High Cpu utilization - Internet Speed drops to very low level

Timothy_Hall — Mon, 04 Jan 2021 13:46:32 GMT

~~That can't be a 15600, a 15600 has 8 physical cores and 16 present with SMT enabled. Your screenshots are showing 32 cores.~~

It appears you have the default 4/28 split and CPUs 0/16 and 1/17 are getting killed with 46% of traffic fully accelerated, while Firewall Worker cores sit relatively idle. Likely that your interfaces are experiencing high frame loss via RX-DRP during the slow periods, which is visible with netstat -ni.

Would recommend decreasing number of firewall instances from 28 to 24 via cpconfig for a target 8/24 split. Once that is done and system rebooted, enable Multi-Queue on your 10Gbps interfaces which will require another separate reboot under Gaia kernel 2.6.18 which appears to be what you are using. Memory and disk look fine.

Re: High Cpu utilization - Internet Speed drops to very low level

Netadmin2020 — Tue, 22 Dec 2020 18:53:42 GMT

You refer that my interfaces are are losing packets during slow periods. What is exactly the reason for this?

These are my setting on perfomance section:

if I choose optimize for packet rate and throughput ? it recommends 6 for performance and 26 for instances.

Re: High Cpu utilization - Internet Speed drops to very low level

PhoneBoy — Wed, 23 Dec 2020 05:55:24 GMT

Personally, I’d upgrade to R80.40 or R81 and enable Dynamic Workloads.
All this stuff will then be tuned automatically on the fly.

Re: High Cpu utilization - Internet Speed drops to very low level

Magnus-Holmberg — Wed, 23 Dec 2020 07:23:05 GMT

in this case reinstalling it to R80.30, 3.10 would have fixed it aswell, but well an upgrade to R80.40 may be easier 🙂

Re: High Cpu utilization - Internet Speed drops to very low level

Timothy_Hall — Wed, 23 Dec 2020 14:06:38 GMT

The reason for the packet loss is insufficient CPU resources allocated to empty the 10Gbps interface ring buffers fast enough via SoftIRQ (RX-DRP), which is a responsibility of an SND/IRQ core. Based on your traffic mix and and the other information you provided, I'd recommend an 8/24 split with Multi-Queue enabled on your busy 10Gbps interfaces.

6/26 as the performance slider is showing is a general recommendation, my 8/24 recommendation is specific based on the info you provided. 6/26 might be OK too.

Re: High Cpu utilization - Internet Speed drops to very low level

Chris_Atkinson — Wed, 23 Dec 2020 14:52:43 GMT

+1 for Timothy Hall's recommendation.

Note also the following regarding Remote Access VPN in case the volumes are significant.
sk165853: High CPU usage on one CPU core when the number of Remote Access users is high

Indeed the number of cores consistent with that of a 15600 is shown here per:
https://www.checkpoint.com/downloads/products/15600-security-gateway-datasheet.pdf

Additionally, note only the 2.6 kernel is supported on the 15600 until R80.40.

Re: High Cpu utilization - Internet Speed drops to very low level

Timothy_Hall — Wed, 23 Dec 2020 15:11:42 GMT

Ah, I wasn't accounting for the "2X" Intel Xeon E5-2630v3 (8/16 core) thought it was just 1X. Thanks for the clarification Chris.

Re: High Cpu utilization - Internet Speed drops to very low level

Netadmin2020 — Thu, 24 Dec 2020 16:38:56 GMT

Ηello again and thank you all for your detailed answers.

I have two nodes, so when I turn the settings at first to the standby node and set the specific interfaces to multi que mode with core split to 6/26 after reboot the cluster seems that is not syncing.

Re: High Cpu utilization - Internet Speed drops to very low level

Timothy_Hall — Thu, 24 Dec 2020 18:08:04 GMT

Correct it will not sync until both members have an identical CoreXL split. You can blunt the impact of a non-stateful failover by temporarily unchecking "Drop out of state TCP packets" on the Stateful Inspection screen of Global Properties and reinstalling policy to both members. Don't forget to recheck it when finished!

Re: High Cpu utilization - Internet Speed drops to very low level

Netadmin2020 — Thu, 24 Dec 2020 18:17:12 GMT

Just to be sure:

a) I turn the second node to multi que and change the core split

b) saving and reboot

c) On the first node I should change the “Drop out of state TCP packets”

d) install policy on both nodes

yes?

Re: High Cpu utilization - Internet Speed drops to very low level

Timothy_Hall — Thu, 24 Dec 2020 18:30:40 GMT

a) uncheck drop out of state packets

b) publish and reinstall to both

c) change core split of second node

d) reboot it (you can't change core split and multi-queue at the same time, you must reboot after doing either one)

e) enable multi queue for relevant interface(s) on second node

f) reboot it

Re: High Cpu utilization - Internet Speed drops to very low level

Netadmin2020 — Thu, 24 Dec 2020 18:45:40 GMT

Great,

with the above steps I suppose that I won’t have any interruptions ?

Re: High Cpu utilization - Internet Speed drops to very low level

PhoneBoy — Thu, 24 Dec 2020 19:16:07 GMT

Something that won’t happen when you use R80.40+ and dynamic workloads to manage the split, FYI.

Re: High Cpu utilization - Internet Speed drops to very low level

Magnus-Holmberg — Thu, 24 Dec 2020 20:46:09 GMT

As the sync is not working, there will be a small dipp.
The packages will need to be reestablished.

Re: High Cpu utilization - Internet Speed drops to very low level

Maarten_Sjouw — Sat, 26 Dec 2020 14:29:56 GMT

Just don't forget to check drop out of state packets at least an hour after you have changed both members and push policy again.

Re: High Cpu utilization - Internet Speed drops to very low level

Netadmin2020 — Sun, 27 Dec 2020 18:03:38 GMT

Merry Christmas to all!

I made a guide and I want to ask some final details.

A) Global Properties uncheck the drop out of state TCP Packets for blunt the impact to the failover
B) Install the policy on both cluster members
C) Change the core split of the Second Node
D) Reboot the Second Node
E) Enable multique at relevant interfaces (2nd node)
F) Reboot

After the above steps theoretical there will be no cluster so I should the reboot the active (first) node and the 2nd node with the new settings will take over?

thanx