High CPU utilization on firewall with two cores

[Glenmark_Impex]

Hello all experts!
We need your advice what we can do for firewall optimization. Currently we are facing performance issue on our firewall. Main issue this is CPU utilization. During working hours we are checking CPUs utilization using cpview and time to time one of CPUs reach 100% of utilization.

Please find current config and "super seven" outputs below.

Enabled features:
FW, Remote Access VPN up to 50 remote users simultaneously, QoS, HTTPS insp, URL and APP filtering, IPS, Threat Prevention IPS, Anti-Bot and Anti-Virus.

[Expert@FW-MSCW-01-01:0]# fwaccel stat
Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #137
Drop Templates : enabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, QOS, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

[Expert@FW-MSCW-01-01:0]# fwaccel stats -s
Accelerated conns/Total conns : 5/4673 (0%)
Delayed conns/(Accelerated conns + PXL conns) : 225696/3136 (7196%)
Accelerated pkts/Total pkts : 12738732/58507915 (21%)
F2Fed pkts/Total pkts : 4980996/58507915 (8%)
PXL pkts/Total pkts : 40788187/58507915 (69%)
QXL pkts/Total pkts : 54107948/58507915 (92%)

[Expert@FW-MSCW-01-01:0]# grep -c ^processor /proc/cpuinfo
8

[Expert@FW-MSCW-01-01:0]# fw ctl affinity -r -l -v
CPU 0: eth0 (irq 67) eth3 (irq 59) eth4 (irq 67) eth7 (irq 59) eth8 (irq 67)
fw_1 fw_3 fw_5
CPU 1: eth1 (irq 75) eth2 (irq 83) eth5 (irq 75) eth6 (irq 83) eth9 (irq 75)
fw_0 fw_2 fw_4
CPU 2:
CPU 3:
CPU 4:
CPU 5:
CPU 6:
CPU 7:
All: rad pepd vpnd mpdaemon in.acapd usrchkd in.msd pdpd in.geod fwpushd rtmd fgd50 fwd lpd cpd cprid
The current license permits the use of CPUs 0, 1 only.

[Expert@FW-MSCW-01-01:0]# /sbin/cpuinfo
HyperThreading=disabled

[Expert@FW-MSCW-01-01:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 15796271 0 0 0 14023053 0 0 0 BMRU
eth1 1500 0 122082 0 0 0 117466 0 0 0 BMRU
eth2 1500 0 3033623 0 0 0 5755418 0 0 0 BMRU
eth3 1500 0 0 0 0 0 0 0 0 0 BMRU
eth4 1500 0 1123043 0 0 0 1533844 0 0 0 BMRU
eth5 1500 0 7958610 0 0 0 11024999 0 0 0 BMRU
eth6 1500 0 27535290 0 0 0 25386131 0 0 0 BMRU
eth7 1500 0 647122 0 0 0 620435 0 0 0 BMRU
eth8 1500 0 19183323 0 0 0 16698501 0 0 0 BMRU
eth8.111 1500 0 3677329 0 0 0 6900712 0 0 0 BMRU
eth8.150 1500 0 13944698 0 0 0 8995520 0 0 0 BMRU
eth8.220 1500 0 522098 0 0 0 650359 0 0 0 BMRU
eth8.230 1500 0 1039134 0 0 0 151978 0 0 0 BMRU
eth9 1500 0 288522 0 0 0 345532 0 0 0 BMRU
lo 16436 0 1440143 0 0 0 1440143 0 0 0 LRU

[Expert@FW-MSCW-01-01:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 1 | 1401 | 1881
1 | Yes | 0 | 908 | 1401
2 | Yes | 1 | 723 | 917
3 | Yes | 0 | 954 | 1417
4 | Yes | 1 | 1211 | 1308
5 | Yes | 0 | 779 | 908

[Expert@FW-MSCW-01-01:0]# cpstat os -f multi_cpu -o 1
Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 100| 0| 100| ?| 10909|
| 2| 0| 59| 41| 59| ?| 10909|
| 3| 0| 0| 100| 0| ?| 10910|
| 4| 0| 0| 100| 0| ?| 10910|
| 5| 0| 0| 100| 0| ?| 10911|
| 6| 1| 3| 96| 4| ?| 10911|
| 7| 5| 3| 91| 9| ?| 10912|
| 8| 0| 1| 99| 1| ?| 10913|
---------------------------------------------------------------------------------

[Expert@FW-MSCW-01-01:0]# free -m
total used free shared buffers cached
Mem: 11877 5778 6098 0 254 2535
-/+ buffers/cache: 2989 8888
Swap: 3067 0 3067

[Expert@FW-MSCW-01-01:0]# cpinfo -y all

This is Check Point CPinfo Build 914000196 for GAIA
[FW1]
HOTFIX_R77_30
HOTFIX_R77_30_JUMBO_HF Take: 351

FW1 build number:
This is Check Point's software version R77.30 - Build 165
kernel: R77.30 - Build 165

[SecurePlatform]
HOTFIX_R77_30_JUMBO_HF Take: 351

[CPinfo]
No hotfixes..

[PPACK]
HOTFIX_R77_30
HOTFIX_R77_30_JUMBO_HF Take: 351

[CVPN]
HOTFIX_R77_30
HOTFIX_R77_30_JUMBO_HF Take: 351

[CPUpdates]
GAIA_WD_UPDATE_SK109359 Take: 0
BUNDLE_R77_30_JUMBO_HF Take: 351

[DIAG]
HOTFIX_R77_30

[G_W_Albrecht]

This version is out of support ! Add 2 more cores and the issue will be resolved.

[Glenmark_Impex]

We are considering increasing CPUs license number. But for now it would be helpful to know will update make our situation with CPUs utilization a little bit easier? 

[G_W_Albrecht · Accepted solution]

Yes, it certainly will ! With 2 cores only, you have no optimization possibilities. Four cores will help much.

[Chris_Atkinson]

"Accept Templates : disabled by Firewall disabled from rule #137"

What does this rule look like in the policy and how many rules are there in total?

[Glenmark_Impex]

Dear Chris.

In total 138 rules, rule 137 contains "traceroute" service and templates were disabled.

But any way we will increase numbers of the CPU cores. 

[Chris_Atkinson]

In R80.10 and above traceroute wouldn't disable templates but given its at the bottom of the current policy it wouldn't have a significant impact here (refer: sk32578).

[Glenmark_Impex]

We have R77.30 Gateway. Update has been planned.

[the_rock]

Guenther is 100% right. Yes, R77.30 is long time out of support, but if you add 2 more cored, you will be fine.

[Timothy_Hall]

Agree with the other posters, your firewall is just very busy for only two cores in an overlapping 2/2 split.  No glaring issues that need to be tuned.  Adding two more cores which will enable a non-overlapping 1/3 default split will make a big difference.

[PhoneBoy]

With a 2 core system running R77.30, there really isn't much tuning you can do to improve performance.
You should upgrade to a supported release and add additional cores.