topic Re: mdps (mgmt plane separation) and corexl in Firewall and Security Management

not expected corexl settings

Luis_Miguel_Mig — Wed, 17 Feb 2021 20:41:05 GMT

I have a cluster with two gateways with 6 cores running r80.40 take 91

Before I configured mdps, 3 cores were running the fw processes and the interfaces configured were running multiqueue as expected.

Then I configured mdps with two interfaces. These two interfaces are configured with cpu "all" and therefore overlap with the cores.

Why not the interfaces configured in the management plane are configure with multiqueue or with a specific cpu not shared with the fw processes?

sim affinity -l
eth0 : All
eth2 : All
eth3 : All
Multi queue interfaces: eth5 eth6

fw ctl affinity -l
Kernel fw_0: CPU 5
Kernel fw_1: CPU 2
Kernel fw_2: CPU 4
Daemon mpdaemon: CPU 2 4 5
Daemon fwd: CPU 2 4 5
Daemon in.acapd: CPU 2 4 5
Daemon lpd: CPU 2 4 5
Daemon in.asessiond: CPU 2 4 5
Daemon vpnd: CPU 2 4 5
Daemon wsdnsd: CPU 2 4 5
Daemon rad: CPU 2 4 5
Daemon usrchkd: CPU 2 4 5
Daemon in.geod: CPU 2 4 5
Daemon cprid: CPU 2 4 5
Daemon cpd: CPU 2 4 5
Interface eth5: has multi queue enabled
Interface eth6: has multi queue enabled

Re: mdps (mgmt plane separation) and corexl

Luis_Miguel_Mig — Wed, 17 Feb 2021 20:40:02 GMT

I have realized that those mgmt interfaces have a different driver "tg3" that doesn't support multiqueue
The affinity configuration is auto, so those interfaces should be set to a free core not to all. Why I am getting these settings? Should I be worried?

cat $FWDIR/conf/fwaffinity.conf
# Process / Interface Affinity Settings
# -------------------------------------
#
# Each line shoud contain:
# 1. A type - 1 character. "i" for interface, "n" for process name, "k" for kernel instance.
# 2. An ID - interface name, process name, or kernel instance number.
# a. For interfaces, you can also write "default", and the setting would apply to any interface not
# mentioned in the file.
# 3. The desired affinity. Either:
# a. One or more CPU numbers.
# b. "all" - all CPUs are eligible.
# c. "ignore" - do nothing for this entry.
# d. "auto" - use any free CPU. A free CPU is one that doesn't appear in any line in this file,
# and doesn't run a worker thread.
#
i default auto

ethtool -i eth3
driver: tg3
version: 3.137
firmware-version: 5719-v1.46 NCSI v1.5.12.0
expansion-rom-version:
bus-info: 0000:02:00.3
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Re: not expected corexl settings

PhoneBoy — Thu, 18 Feb 2021 03:47:28 GMT

MDPS requires a minimum of 8 cores, and you say you have 6.
That may be why it’s not working.

Re: not expected corexl settings

Aviad_Hadarian — Thu, 18 Feb 2021 06:26:02 GMT

Hi @Luis_Miguel_Mig 'All' simply means it can use any of the CPU (only one can be really used)

Can you share the output of 'mq_mng -o'?

Re: not expected corexl settings

Aviad_Hadarian — Thu, 18 Feb 2021 06:26:22 GMT

@PhoneBoy the limitation is 4 CPU's and 3 firewall instances

Re: not expected corexl settings

Luis_Miguel_Mig — Thu, 18 Feb 2021 11:39:36 GMT

definition of auto and all at $FWDIR/conf/fwaffinity.conf

# b. "all" - all CPUs are eligible.
# c. "ignore" - do nothing for this entry.
# d. "auto" - use any free CPU. A free CPU is one that doesn't appear in any line in this file,

healthcheck script (sk121447) gives me a warning about overlapping workers and SNDs - meaning it is not good I guess

In R77, when the affinity is set to auto, the output of "fw ctl affinity -l -r " shows specific cores for my interfaces, not all.

[Expert@fw1:1]# clish -c "show configuration" | grep "mdps int"
set mdps interface bond0 sync on
set mdps interface eth3 management on
[Expert@fw1:1]# mq_mng -o
No multiqueue supported interfaces available
[Expert@fw1:1]# dplane
Context set to Data Plane
[Expert@fw1:0]# mq_mng -o
Total 6 cores. Multiqueue 3 cores
i/f type state mode cores
------------------------------------------------------------------------------------------------
eth5 ixgbe Up Auto (3/3) 0,3,1
eth6 ixgbe Up Auto (3/3) 0,3,1

output from the healthcheck script

+-----------------------+
| CoreXL |
+-----------------------+
CoreXL Notice: Cores detected operating as both fw workers and SNDs. Please review sk98737 and sk98348 for more information.
CoreXL Settings:
Interface eth0: CPU all
Interface eth2: CPU all
Interface eth3: CPU all
Kernel fw_0: CPU 5
Kernel fw_1: CPU 2
Kernel fw_2: CPU 4
Interface eth5: has multi queue enabled
Interface eth6: has multi queue enabled