High CPU

flachance — Fri, 14 Oct 2022 15:23:23 GMT

Hi, we're experiencing some high cpu usage and to be honest I'm not sure what to make of the cpview results. What is PM tier 1? Is it related to IPS?

Re: High CPU

flachance — Fri, 14 Oct 2022 15:58:20 GMT

I tried turning off ips but it didn't help

Re: High CPU

PhoneBoy — Fri, 14 Oct 2022 16:17:42 GMT

Pattern Matcher is used with any blade that is not Firewall and VPN.
That would include but isn't necessarily caused by IPS.
Please provide the output of the Super 7 Commands: https://community.checkpoint.com/t5/Scripts/S7PAC-Super-Seven-Performance-Assessment-Commands/m-p/40528#M703

Re: High CPU

flachance — Fri, 14 Oct 2022 16:38:22 GMT

Accept Templates : disabled by Firewall
Layer CIRB incoming disables template offloads from rule #3
Throughput acceleration still enabled.
Layer RZ Inbound disables template offloads from rule #1
Throughput acceleration still enabled.
Layer OnPrem2AzureInfrastructure disables template offloads from rule #2
Throughput acceleration still enabled.
Layer Mtl2OnPremRZ disables template offloads from rule #1
Throughput acceleration still enabled.
Layer Azure2OnPrem RZ disables template offloads from rule #1
Throughput acceleration still enabled.
Layer RZ-OnPrem&Azure disables template offloads from rule #8
Throughput acceleration still enabled.
Drop Templates : enabled
NAT Templates : disabled by Firewall
Layer CIRB incoming disables template offloads from rule #3
Throughput acceleration still enabled.
Layer RZ Inbound disables template offloads from rule #1
Throughput acceleration still enabled.
Layer OnPrem2AzureInfrastructure disables template offloads from rule #2
Throughput acceleration still enabled.
Layer Mtl2OnPremRZ disables template offloads from rule #1
Throughput acceleration still enabled.
Layer Azure2OnPrem RZ disables template offloads from rule #1
Throughput acceleration still enabled.
Layer RZ-OnPrem&Azure disables template offloads from rule #8
Throughput acceleration still enabled.

+-----------------------------------------------------------------------------+
| Command #2: fwaccel stats -s |
| |
| Check for : Accelerated conns/Totals conns: >25% good, >50% great |
| Accelerated pkts/Total pkts : >50% great |
| PXL pkts/Total pkts : >50% OK |
| F2Fed pkts/Total pkts : <30% good, <10% great |
| |
| Chapter 9: SecureXL throughput acceleration |
| Page 287, Packet/Throughput Acceleration: The Three Kernel Paths |
+-----------------------------------------------------------------------------+
| Output: |
Accelerated conns/Total conns : 46/31721 (0%)
Accelerated pkts/Total pkts : 171313823924/173420969516 (98%)
F2Fed pkts/Total pkts : 2107145592/173420969516 (1%)
F2V pkts/Total pkts : 347523193/173420969516 (0%)
CPASXL pkts/Total pkts : 51117776512/173420969516 (29%)
PSLXL pkts/Total pkts : 118964632574/173420969516 (68%)
CPAS pipeline pkts/Total pkts : 0/173420969516 (0%)
PSL pipeline pkts/Total pkts : 0/173420969516 (0%)
CPAS inline pkts/Total pkts : 0/173420969516 (0%)
PSL inline pkts/Total pkts : 0/173420969516 (0%)
QOS inbound pkts/Total pkts : 0/173420969516 (0%)
QOS outbound pkts/Total pkts : 0/173420969516 (0%)
Corrected pkts/Total pkts : 0/173420969516 (0%)

+-----------------------------------------------------------------------------+
| Command #3: grep -c ^processor /proc/cpuinfo && /sbin/cpuinfo |
| |
| Check for : If number of cores is roughly double what you are excpecting, |
| hyperthreading may be enabled |
| |
| Chapter 7: CoreXL Tuning |
| Page 239 |
+-----------------------------------------------------------------------------+
| Output: |
8
HyperThreading=disabled

+-----------------------------------------------------------------------------+
| Command #4: fw ctl affinity -l -r |
| |
| Check for : SND/IRQ/Dispatcher Cores, # of CPU's allocated to interface(s) |
| Firewall Workers/INSPECT Cores, # of CPU's allocated to fw_x |
| R77.30: Support processes executed on ALL CPU's |
| R80.xx: Support processes only executed on Firewall Worker Cores|
| |
| Chapter 7: CoreXL Tuning |
| Page 221 |
+-----------------------------------------------------------------------------+
| Output: |
CPU 0: eth1 eth2 eth3
CPU 1: fw_5
fwd lpd in.acapd mta_monitor in.asessiond usrchkd dtpsd pepd mpdaemon vpnd wsdnsd rtmd cprid fwpushd in.emaild.mta rad dtlsd core_uploader pdpd in.geod cpd cprid
CPU 2: fw_3
fwd lpd in.acapd mta_monitor in.asessiond usrchkd dtpsd pepd mpdaemon vpnd wsdnsd rtmd cprid fwpushd in.emaild.mta rad dtlsd core_uploader pdpd in.geod cpd cprid
CPU 3: fw_1
fwd lpd in.acapd mta_monitor in.asessiond usrchkd dtpsd pepd mpdaemon vpnd wsdnsd rtmd cprid fwpushd in.emaild.mta rad dtlsd core_uploader pdpd in.geod cpd cprid
CPU 4:
CPU 5: fw_4
fwd lpd in.acapd mta_monitor in.asessiond usrchkd dtpsd pepd mpdaemon vpnd wsdnsd rtmd cprid fwpushd in.emaild.mta rad dtlsd core_uploader pdpd in.geod cpd cprid
CPU 6: fw_2
fwd lpd in.acapd mta_monitor in.asessiond usrchkd dtpsd pepd mpdaemon vpnd wsdnsd rtmd cprid fwpushd in.emaild.mta rad dtlsd core_uploader pdpd in.geod cpd cprid
CPU 7: fw_0
fwd lpd in.acapd mta_monitor in.asessiond usrchkd dtpsd pepd mpdaemon vpnd wsdnsd rtmd cprid fwpushd in.emaild.mta rad dtlsd core_uploader pdpd in.geod cpd cprid
All:
Interface eth8: has multi queue enabled
Interface eth9: has multi queue enabled

+-----------------------------------------------------------------------------+
| Command #5: netstat -ni |
| |
| Check for : RX/TX errors |
| RX-DRP % should be <0.1% calculated by (RX-DRP/RX-OK)*100 |
| TX-ERR might indicate Fast Ethernet/100Mbps Duplex Mismatch |
| |
| Chapter 2: Layers 1&2 Performance Optimization |
| Page 28-35 |
| |
| Chapter 7: CoreXL Tuning |
| Page 204 |
| Page 206 (Network Buffering Misses) |
+-----------------------------------------------------------------------------+
| Output: |
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
bond1 1500 0 131205881921 0 0 0 98197788926 0 0 0 BMmRU
bond1.2 1500 0 11075878 0 49750 0 19263187 0 0 0 BMRU
bond1.11 1500 0 10190083 0 156 0 9717573 0 0 0 BMRU
bond1.100 1500 0 77510858121 0 1307019 0 78822159513 0 0 0 BMRU
bond1.106 1500 0 4380993 0 379761 0 235075028 0 0 0 BMRU
bond1.108 1500 0 17092053715 0 183806 0 5674536270 0 0 0 BMRU
bond1.112 1500 0 21332727560 0 823 0 5877593652 0 0 0 BMRU
bond1.140 1500 0 2742324 0 917 0 2677623 0 0 0 BMRU
bond1.150 1500 0 3164913167 0 59 0 642979284 0 0 0 BMRU
bond1.152 1500 0 192985 0 0 0 149949 0 0 0 BMRU
bond1.160 1500 0 827531034 0 24259 0 229120864 0 0 0 BMRU
bond1.170 1500 0 13975269 0 31 0 12763680 0 0 0 BMRU
bond1.171 1500 0 19 0 0 0 42795 0 0 0 BMRU
bond1.355 1500 0 5484716373 0 0 0 4311281028 0 0 0 BMRU
bond1.500 1500 0 3778665907 0 2438 0 1726948704 0 0 0 BMRU
bond1.550 1500 0 565614091 0 17 0 192662881 0 0 0 BMRU
bond1.560 1500 0 585068115 0 202 0 215526710 0 0 0 BMRU
bond1.570 1500 0 815696026 0 358 0 238497128 0 0 0 BMRU
bond1.804 1500 0 5303533 0 0 0 9059566 0 0 0 BMRU
eth1 1500 0 7191113 0 6 0 8530266 0 0 0 BMRU
eth2 1500 0 42402513961 0 321122427 0 114851655091 0 0 0 BMRU
eth3 1500 0 559319557 0 333462 0 546147561 0 0 0 BMRU
eth8 1500 0 63152389162 0 0 0 13837171475 0 0 0 BMsRU
eth9 1500 0 68053487950 0 0 0 84360614272 0 0 0 BMsRU
lo 65536 0 81592231 0 0 0 81592231 0 0 0 ALMPRU

interface eth1: [32mThere were no RX drops in the past 0.5 seconds(B[m
interface eth1 rx_missed_errors :
interface eth1 rx_fifo_errors :
interface eth1 rx_no_buffer_count:

interface eth2: [32mThere were no RX drops in the past 0.5 seconds(B[m
interface eth2 rx_missed_errors :
interface eth2 rx_fifo_errors :
interface eth2 rx_no_buffer_count:

interface eth3: [32mThere were no RX drops in the past 0.5 seconds(B[m
interface eth3 rx_missed_errors :
interface eth3 rx_fifo_errors :
interface eth3 rx_no_buffer_count:

interface eth8: [32mThere were no RX drops in the past 0.5 seconds(B[m
interface eth8 rx_missed_errors : 0
interface eth8 rx_fifo_errors : 0
interface eth8 rx_no_buffer_count: 0

interface eth9: [32mThere were no RX drops in the past 0.5 seconds(B[m
interface eth9 rx_missed_errors : 0
interface eth9 rx_fifo_errors : 0
interface eth9 rx_no_buffer_count: 0

+-----------------------------------------------------------------------------+
| Command #6: fw ctl multik stat |
| |
| Check for : Large # of conns on Worker 0 - IPSec VPN/VoIP? |
| Large imbalance of connections on a single or multiple Workers |
| |
| Chapter 7: CoreXL Tuning |
| Page 241 |
| |
| Chapter 8: CoreXL VPN Optimization |
| Page 256 |
+-----------------------------------------------------------------------------+
| Output: |
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 7 | 8073 | 11972
1 | Yes | 3 | 2891 | 9650
2 | Yes | 6 | 7990 | 10876
3 | Yes | 2 | 7732 | 10735
4 | Yes | 5 | 2764 | 8529
5 | Yes | 1 | 4610 | 9798

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 76| 24| 76| ?| 42582|
| 2| 1| 99| 1| 99| ?| 42582|
| 3| 7| 58| 35| 65| ?| 42582|
| 4| 0| 100| 0| 100| ?| 42582|
| 5| 0| 8| 92| 8| ?| 42582|
| 6| 0| 100| 0| 100| ?| 42582|
| 7| 5| 62| 33| 67| ?| 42582|
| 8| 4| 70| 26| 74| ?| 42582|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 78| 22| 78| ?| 87287|
| 2| 0| 100| 0| 100| ?| 43644|
| 3| 12| 49| 39| 61| ?| 87290|
| 4| 0| 100| 0| 100| ?| 43646|
| 5| 0| 7| 93| 7| ?| 87296|
| 6| 0| 100| 0| 100| ?| 43648|
| 7| 2| 79| 19| 81| ?| 87299|
| 8| 3| 67| 30| 70| ?| 87303|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 78| 22| 78| ?| 84875|
| 2| 0| 100| 0| 100| ?| 42437|
| 3| 7| 55| 38| 62| ?| 84873|
| 4| 0| 100| 0| 100| ?| 42438|
| 5| 0| 7| 93| 7| ?| 84875|
| 6| 0| 100| 0| 100| ?| 42437|
| 7| 1| 79| 20| 80| ?| 84883|
| 8| 3| 60| 37| 63| ?| 84882|
---------------------------------------------------------------------------------

+-----------------------------------------------------------------------------+
| Thanks for using s7pac |
+-----------------------------------------------------------------------------+

Re: High CPU

flachance — Fri, 14 Oct 2022 16:56:06 GMT

Re: High CPU

PhoneBoy — Fri, 14 Oct 2022 17:54:45 GMT

This is a bit concerning:

That said, most of your packets are accelerated, so this may not be a big deal.
You do have a large number of receive drops on your bond interface and on eth2, which might be something worth investigating.

What version/JHF are we working with here on what precise hardware?
output of enabled_blades would also be helpful to further contextualize this.

Also, you have 10 connections that are taking 92% of the CPU.
What precisely are these connections for and what precise policies relate to them?
Depending on what they are (and if they are trusted), we can fully accelerate them using fast_accel to reduce the overall CPU load.

Re: High CPU

flachance — Fri, 14 Oct 2022 18:03:05 GMT

I looked at the mentioned layers and rule numbers where it says disables template offload but I don't see/notice anything particular about them.

We're running R80.40 JHF 173 on a pair of HP Proliant servers. The connections taking all that CPU are from the backup server so they would be trusted. Maybe it's always been like that and I just happened to notice it today but I don't think so. Is it easy to fully accelerate the connections from that server?

Re: High CPU

flachance — Fri, 14 Oct 2022 18:15:32 GMT

once added should it be effective immediatly or just on new connections?

Re: High CPU

PhoneBoy — Fri, 14 Oct 2022 18:23:27 GMT

Given how SecureXL works in R80.20 and above, I believe it should be effective immediately.
However, it might require a new connection to be established.

topic Re: High CPU in Firewall and Security Management

High CPU

Re: High CPU

Re: High CPU

Re: High CPU

Re: High CPU

Re: High CPU

Re: High CPU

Re: High CPU

Re: High CPU

Re: High CPU