CoreXL_FW 3 CPUs@100%

[Daniel_Kavan]

Hi,

I noticed that I have 3 CPUs all in the red, at 100%

[Expert@gw2:0]# fw ctl affinity -l -r
CPU 0:
CPU 1: eth4 eth5 eth6 eth7 eth10 eth11
CPU 2:
CPU 3: fw_1
mpdaemon fwd lpd rad vpnd wsdnsd usrchkd pepd in.asessiond fwpushd pdpd in.acapd cprid cpd
CPU 4:
CPU 5:
CPU 6:
CPU 7:
CPU 8:
CPU 9: eth0
CPU 10: fw_2
mpdaemon fwd lpd rad vpnd wsdnsd usrchkd pepd in.asessiond fwpushd pdpd in.acapd cprid cpd
CPU 11: fw_0
mpdaemon fwd lpd rad vpnd wsdnsd usrchkd pepd in.asessiond fwpushd pdpd in.acapd cprid cpd
CPU 12:
CPU 13:
CPU 14:
CPU 15:
All:
The current license permits the use of CPUs 0, 1, 2, 3, 8, 9, 10, 11 only.

CPVIEW CPU shows CoreXL_FW using 3 CPUs at 100%

CorXL_SND is using 2 @ 2%

Other 11 0%

So, I'm just using 5 of my CPUs.   It looks like I have 3 more I can use.

My load on this box has been at 20% so I was suspecting some thing like this.

[PhoneBoy]

Assuming you're licensed for 8 cores, you should have a 2/6 split (2 SND cores, 6 worker/fw instances), at least by default. 
Which means you should go into cpconfig and change the number of firewall instances to 6.
This will require the same setting on both cluster members and require a reboot.

[Daniel_Kavan]

4 cores... so 1/3 split would be right.  I guess I just need to make it more efficient.  This gw is on R80.40 JHF94.  R81 may be better.

[PhoneBoy]

The fact you have two cores used for interfaces is...interesting.
I would strongly consider adding more cores to your Open Server license also.

[Timothy_Hall]

Perhaps I'm reading it wrong, but it looks like there are 4 physical cores licensed, but you are getting 8 (0, 1, 2, 3, 8, 9, 10, 11) due to SMT being enabled.  It looks like things started with the default 1/3 split with SMT disabled, then SMT got enabled and now you have basically a 5/3 split and the three worker/instance cores are getting killed.  You must be using R80.30 or earlier with the 2.6.18 kernel which is why Multi-Queue is disabled (or you are using crappy Broadcom or other off-brand NICs which don't support Multi-Queue).

I'd need to see full Super Seven outputs to make a good recommendation, but based on what I can see here so far you should probably run cpconfig and set number of instances to 6 for a 2/6 split.  That should help a lot.

[Daniel_Kavan]

R80.40 JHF94.

CoreXL is currently enabled with 3 IPv4 firewall instances and 3 IPv6 firewall instances.

eth0 : All
eth10 : All
eth11 : All
eth4 : All
eth5 : All
eth6 : All
eth7 : All
[Expert@gw2:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 11 | 2378 | 8371
1 | Yes | 3 | 2393 | 8373

2 | Yes | 10 | 2391 | 8500

load average: 4.32, 4.41, 4.47

[HeikoAnkenbrand · Accepted solution]

Hi @Daniel_Kavan,

Regarding your statement:
CUT>>>
CPVIEW CPU shows CoreXL_FW using 3 CPUs at 100%
CorXL_SND is using 2 @ 2%
<<<CUT

Exciting I find the distribution 2/3 🙂
This is not possible with a 4 core license.
If you have a 8 core license, not all cores are used and the default distribution should be 2/6.
I would clean up this first.

If you have a 4 core license, you should only use  - in your case - 1 x SND and  3 x CoreXL.
Problem is that you need to reduce the load on the CoreXL instances.
To do that, you need to reduce the PSL,PSLXL and CPAS traffic.
More here R8x - Security Gateway Architecture (Content Inspection).
This means, for example, less IPS, AntiBot,... analysis.
Here you can - for example - exclude internal networks from IPS, AntiBot, AV... unfortunately, this reduced the security level.

Sometimes you can also set the CPU itself to the highest clock speed in the BIOS. 
This also helps in some cases on a open server to have more CPU performance and thus more CoreXL performance.
More read here R8x - Performance Tuning Tip - BIOS.

Alternatively, you can buy a larger license (8 cores) and use a 2/6 distribution.

PS:
I would always install the latest Jumbo hotfix sk165456 at the moment R80.40 jumbo HF 118.

[Daniel_Kavan]

Thanks everyone. 

Is my gateway going to 2 SND automatically, because of the load?   If so, is there a way to knock it back to one SND in cpconfig?

Update: removing Anti Bot blade from the gw and removing some of the internal networks did knock me down to 1 SND being used.  However, for now my load is higher than ever, over 5 and I'm still at 100% for all 3 CPUs.  The User % is what seems extraordinarily high, see attached.

If it's automatic, then it sounds like I need to reduce the load, adjust the clock speed, consider a new license, & update to 118 as described by HA.

uname -a
Linux gw2 3.10.0-957.21.3cpx86_64 #1 SMP Thu Mar 4 12:39:55 IST 2021 x86_64 x86_64 x86_64 GNU/Linux

This is NOT a cluster, not using clusterXL

[Daniel_Kavan]

fw ctl affinity -l -r
CPU 0: eth4 eth0 eth5 eth6 eth7 eth10 eth11
CPU 1: eth4 eth0 eth5 eth6 eth7 eth10 eth11
CPU 2: eth4 eth0 eth5 eth6 eth7 eth10 eth11
CPU 3: eth4 eth0 eth5 eth6 eth7 eth10 eth11
fw_1
mpdaemon fwd lpd rad vpnd wsdnsd usrchkd pepd in.asessiond fwpushd pdpd in.acapd cprid cpd
CPU 4: eth4 eth0 eth5 eth6 eth7 eth10 eth11
CPU 5: eth4 eth0 eth5 eth6 eth7 eth10 eth11
CPU 6: eth4 eth0 eth5 eth6 eth7 eth10 eth11
CPU 7: eth4 eth0 eth5 eth6 eth7 eth10 eth11
CPU 8:
CPU 9:
CPU 10: fw_2
mpdaemon fwd lpd rad vpnd wsdnsd usrchkd pepd in.asessiond fwpushd pdpd in.acapd cprid cpd
CPU 11: fw_0
mpdaemon fwd lpd rad vpnd wsdnsd usrchkd pepd in.asessiond fwpushd pdpd in.acapd cprid cpd
CPU 12:
CPU 13:
CPU 14:
CPU 15:
All:
The current license permits the use of CPUs 0, 1, 2, 3, 8, 9, 10, 11 only.

[Wolfgang]

@Daniel_Kavan one more thing to mention...

Set the number of CPUs via the BIOS of your open server according to your license.

8 core license => 8 CPUs.

4 core license => 4 CPUs.

It's confusing for the system running more cores then licensed. There are some some known strange behaviours wiht such a mismatch.

[HeikoAnkenbrand]

I think there is something wrong with the license macro file or there was an Eval license installed, then it also comes to these effects. What does a "cplic print -x" show?

[Daniel_Kavan]

Host Expiration Signature Features
10.ext.IP.of.gw.111 never aUNquevYRrMjLTHJbRnEm3AVTS846GnE53Vk CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
10.manager.10 never aHtzCE5T2AZMRgLcYZe5gRRzisQWZDhWvysu CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918

Contract Coverage:

# ID Expiration SKU
===+===========+============+====================
1 | PYHD61S | 30Nov2019 | CPSB-EBP-TE
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
2 | 3SWY3P6 | 30Nov2021 | CPSB-EBP-URLF
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
3 | 43O71G5 | 30Nov2019 | CPSB-EBP-TEX
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
4 | I0DAGHR | 30Nov2021 | CPSB-EBP-AV
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
5 | 48O2ASU | 30Nov2021 | CPSB-EBP-APCL
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
6 | 4HF6148 | 30Nov2021 | CPSB-EBP-CTNT
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
7 | TPH953O | 30Nov2021 | CPES-SS-STANDARD
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
8 | I0Y0YR3 | 30Nov2021 | CPSB-EBP-ABOT
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
9 | U5UDPS7 | 30Nov2021 | CPSB-EBP-IPS
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918
===+===========+============+====================
10 | 55D7Y58 | 30Nov2021 | CPSB-EBP-ASPM
+-----------+------------+--------------------
|Covers: CPSB-ADN-M CPSB-ACCL-M CPVP-SNX-100-NGX CPVP-SNX-100-NGX CPSB-SWB CK-E823052B5918
| CPSG-C-4-U CPSB-FW CPSB-VPN CPSB-ADN CPSB-ACCL CPSB-SSLVPN-200 CPSB-IA CPSB-IPS CPSB-APCL CPSB-ABOT-M CPSB-ASPM CPSB-AV CPSB-URLF CPSB-CTNT CK-E823052B5918

[Daniel_Kavan]

Also, I have the SAME hardware running with R81 on another gw, not having this issue.

[PhoneBoy]

Does this Open Server appliance have HyperThreading enabled?
One of the reasons we initially held back support of Open Servers on R80.40 was related to HyperThreading.
From a licensing point of view, we treat HyperThreaded cores the same as physical ones. 
This bug sounds...related to that and I recommend a TAC case.

[Timothy_Hall]

Agreed, I speculated in an earlier post that the odd-looking CoreXL split config had something to do with SMT.

[Daniel_Kavan · Accepted solution]

TAC fixed this quickly.  There was a bug with MAB and httpd running.   Thanks for all the good advice, including performance optimization with open server & SMT.