topic Re: r80.40 - VSX interface affinity HP DL380G10 Openserver tg3 in Firewall and Security Management

r80.40 - VSX interface affinity HP DL380G10 Openserver tg3

jkougoulos — Mon, 04 Apr 2022 18:11:11 GMT

Hello,

I am new in Checkpoint firewalls (well I configured a couple of rules 20 years ago in Solaris based one, but this does not count) and I have inherited a VSX HA cluster running on HP DL380 Gen10 Openserver with 3 quad tg3 based (broadcom) 1G NICs and a couple of dual Intel 10G NICs. The box has 2x10 cores, the first 8 of them are licensed.

I have read a lot of documents and skXXXXX in the last days, many forum posts and have done several experiments. I still have some issues with setting up the affinity of the interfaces properly.

Long story short, it looks like each tg3 interface is serviced by 5 IRQs

eg:

# grep "eth0-" /proc/interrupts | awk ' { print $1, $23 } '
218: eth0-0
219: eth0-txrx-1
220: eth0-tx-2
221: eth0-tx-3
222: eth0-tx-4

Now, when I setup the affinity of the interface eth0 to CPU 0, this is what I get:

# fw ctl affinity -s -i eth0 0
Interface eth0: CPU 0 - set successfully
Multi-queue affinity was not changed. For More info, see sk113834.

# fw ctl affinity -l -r -v -a
CPU 0: eth0 (irq 222) eth4 (irq 237) eth5 (irq 227)
CPU 1: eth0 (irq 222) eth1 (irq 267) eth2 (irq 242) eth3 (irq 247) eth5 (irq 227) eth6 (irq 217) eth7 (irq 257) eth11 (irq 212) eth12 (irq 267) eth13 (irq 232) eth14 (irq 252)
CPU 2: eth0 (irq 222) eth5 (irq 227)
CPU 3: eth0 (irq 222) eth5 (irq 227)
CPU 4: eth0 (irq 222) eth5 (irq 227)
CPU 5: eth0 (irq 222) eth5 (irq 227)
CPU 6: eth0 (irq 222) eth5 (irq 227)
CPU 7: eth0 (irq 222) eth5 (irq 227)
CPU 8: eth0 (irq 222) eth5 (irq 227)
CPU 9: eth0 (irq 222) eth5 (irq 227)
CPU 10:
CPU 11:
CPU 12:
CPU 13:
CPU 14:
CPU 15:
CPU 16:
CPU 17:
CPU 18:
CPU 19:
All:
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.
Interface eth8: has multi queue enabled
Interface eth9: has multi queue enabled

# fw ctl affinity -l
Interface eth0: CPU 0 1 2 3 4 5 6 7 8 9
Interface eth1: CPU 1
Interface eth2: CPU 1
Interface eth3: CPU 1
Interface eth4: CPU 0
Interface eth5: CPU 0 1 2 3 4 5 6 7 8 9
Interface eth6: CPU 1
Interface eth7: CPU 1
Interface eth11: CPU 1
Interface eth12: CPU 1
Interface eth13: CPU 1
Interface eth14: CPU 1
VS_0: CPU 2 3 4 5 6 7
VS_0 fwk: CPU 2 3 4 5 6 7
VS_1: CPU 2 3 4 5 6 7
VS_1 fwk: CPU 2 3 4 5 6 7
VS_5: CPU 2 3 4 5 6 7
VS_5 fwk: CPU 2 3 4 5 6 7
VS_8: CPU 2 3 4 5 6 7
VS_8 fwk: CPU 2 3 4 5 6 7
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.
Interface eth8: has multi queue enabled
Interface eth9: has multi queue enabled

Now if I go a bit deeper:

# for i in `grep "eth0-" /proc/interrupts | awk -F: ' { print $1 } ' | sed 's/://' `; do echo -n "$i -> " ; cat /proc/irq/$i/smp_affinity ; done
218 -> 00001
219 -> 003ff
220 -> 003ff
221 -> 003ff
222 -> 003ff

Notice also that in the output of fw ctl affinity -l -r -v -a, eth1 seems to use the same IRQ with eth12.

While, in reality:

# grep "eth1-" /proc/interrupts | awk ' { print $1, $23 } '
258: eth1-0
259: eth1-txrx-1
260: eth1-tx-2
261: eth1-tx-3
262: eth1-tx-4

# grep "eth12-" /proc/interrupts | awk ' { print $1, $23 } '
263: eth12-0
264: eth12-txrx-1
265: eth12-tx-2
266: eth12-tx-3
267: eth12-tx-4

So, my question is... does it look the same in other installations using tg3 driver in r80.40 / is it expected behaviour or should I look for something else?

Thanks!

John

Re: r80.40 - VSX interface affinity HP DL380G10 Openserver tg3

G_W_Albrecht — Tue, 05 Apr 2022 15:32:13 GMT

The issue you have should be fixed in R80.40 sk166356: 'fw ctl affinity' and "sim affinity" commands showing wrong data

Re: r80.40 - VSX interface affinity HP DL380G10 Openserver tg3

Wolfgang — Tue, 05 Apr 2022 15:45:31 GMT

@jkougoulos one more thing to observe.... disable your unlicensed cores in the BIOS of your open server hardware. There are some known strange problems if more cores active then licensed.

Re: r80.40 - VSX interface affinity HP DL380G10 Openserver tg3

jkougoulos — Wed, 06 Apr 2022 07:54:30 GMT

@Wolfgangthanks for this hint. I saw somewhere that tg3 may enable more channels depending on number of cores, but if I want to reduce the cores to 8 in Bios, probably this would mean to switch to 2x4 cores - moving the load between the 2 processors which might have some performance impact.

I will start with ethtool tuning first and if it plays any role and see if I need to resort to reducing the number of cores. I guess I could also set the affinity manually using commands to /proc/irq/xxx/smp_affinity

Re: r80.40 - VSX interface affinity HP DL380G10 Openserver tg3

jkougoulos — Wed, 06 Apr 2022 08:22:07 GMT

thanks for the pointers!

sk166356 looks similar but a bit different case. Now that I look closer, I believe I can emulate the way that fw ctl affinity matches the interfaces to interrupts (maybe it is the "last resort" method):

# grep eth1 /proc/interrupts | tail -1 | awk ' { print $1 } ' 267: # grep eth12 /proc/interrupts | tail -1 | awk ' { print $1 } ' 267:

for sk171526, I believe this installation started with a clean r80.40. Also this relates to FWK affinity, I don't see a problem there:

# fw ctl affinity -l -v Interface eth0 (irq 222): CPU 0 1 2 3 4 5 6 7 8 9 Interface eth1 (irq 267): CPU 1 Interface eth2 (irq 242): CPU 1 Interface eth3 (irq 247): CPU 1 Interface eth4 (irq 237): CPU 0 Interface eth5 (irq 227): CPU 0 1 2 3 4 5 6 7 8 9 Interface eth6 (irq 217): CPU 1 Interface eth7 (irq 257): CPU 1 Interface eth11 (irq 212): CPU 1 Interface eth12 (irq 267): CPU 1 Interface eth13 (irq 232): CPU 1 Interface eth14 (irq 252): CPU 1 VS_0: CPU 2 3 4 5 6 7 VS_0 fwk: CPU 2 3 4 5 6 7 VS_1: CPU 2 3 4 5 6 7 VS_1 fwk: CPU 2 3 4 5 6 7 VS_5: CPU 2 3 4 5 6 7 VS_5 fwk: CPU 2 3 4 5 6 7 VS_8: CPU 2 3 4 5 6 7 VS_8 fwk: CPU 2 3 4 5 6 7 The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only. Interface eth8: has multi queue enabled Interface eth9: has multi queue enabled # fw ctl multik stat ID | Active | CPU | Connections | Peak ---------------------------------------------- 0 | Yes | 2-7 | 26 | 212 1 | Yes | 2-7 | 20 | 212 2 | Yes | 2-7 | 23 | 212 3 | Yes | 2-7 | 24 | 213 4 | Yes | 2-7 | 26 | 212 5 | Yes | 2-7 | 24 | 213