r80.40 - VSX interface affinity HP DL380G10 Openserver tg3

[jkougoulos]

Hello,

I am new in Checkpoint firewalls (well I configured a couple of rules 20 years ago in Solaris based one, but this does not count) and I have inherited a VSX HA cluster running on HP DL380 Gen10 Openserver with 3 quad tg3 based (broadcom) 1G NICs and a couple of dual Intel 10G NICs. The box has 2x10 cores, the first 8 of them are licensed.

I have read a lot of documents and skXXXXX in the last days, many forum posts and have done several experiments. I still have some issues with setting up the affinity of the interfaces properly.

Long story short, it looks like each tg3 interface is serviced by 5 IRQs

eg:

Click to Expand

# grep "eth0-" /proc/interrupts | awk ' { print $1, $23 } '
218: eth0-0
219: eth0-txrx-1
220: eth0-tx-2
221: eth0-tx-3
222: eth0-tx-4

Now, when I setup the affinity of the interface eth0 to CPU 0, this is what I get:

Click to Expand

# fw ctl affinity -s -i eth0 0
Interface eth0: CPU 0 - set successfully
Multi-queue affinity was not changed. For More info, see sk113834.


# fw ctl affinity -l -r -v -a
CPU 0: eth0 (irq 222) eth4 (irq 237) eth5 (irq 227)
CPU 1: eth0 (irq 222) eth1 (irq 267) eth2 (irq 242) eth3 (irq 247) eth5 (irq 227) eth6 (irq 217) eth7 (irq 257) eth11 (irq 212) eth12 (irq 267) eth13 (irq 232) eth14 (irq 252)
CPU 2: eth0 (irq 222) eth5 (irq 227)
CPU 3: eth0 (irq 222) eth5 (irq 227)
CPU 4: eth0 (irq 222) eth5 (irq 227)
CPU 5: eth0 (irq 222) eth5 (irq 227)
CPU 6: eth0 (irq 222) eth5 (irq 227)
CPU 7: eth0 (irq 222) eth5 (irq 227)
CPU 8: eth0 (irq 222) eth5 (irq 227)
CPU 9: eth0 (irq 222) eth5 (irq 227)
CPU 10:
CPU 11:
CPU 12:
CPU 13:
CPU 14:
CPU 15:
CPU 16:
CPU 17:
CPU 18:
CPU 19:
All:
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.
Interface eth8: has multi queue enabled
Interface eth9: has multi queue enabled



# fw ctl affinity -l
Interface eth0: CPU 0 1 2 3 4 5 6 7 8 9
Interface eth1: CPU 1
Interface eth2: CPU 1
Interface eth3: CPU 1
Interface eth4: CPU 0
Interface eth5: CPU 0 1 2 3 4 5 6 7 8 9
Interface eth6: CPU 1
Interface eth7: CPU 1
Interface eth11: CPU 1
Interface eth12: CPU 1
Interface eth13: CPU 1
Interface eth14: CPU 1
VS_0: CPU 2 3 4 5 6 7
VS_0 fwk: CPU 2 3 4 5 6 7
VS_1: CPU 2 3 4 5 6 7
VS_1 fwk: CPU 2 3 4 5 6 7
VS_5: CPU 2 3 4 5 6 7
VS_5 fwk: CPU 2 3 4 5 6 7
VS_8: CPU 2 3 4 5 6 7
VS_8 fwk: CPU 2 3 4 5 6 7
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.
Interface eth8: has multi queue enabled
Interface eth9: has multi queue enabled

Now if I go a bit deeper:

Click to Expand

# for i in `grep "eth0-" /proc/interrupts | awk -F: ' { print $1 } ' | sed 's/://' `; do echo -n "$i -> " ; cat /proc/irq/$i/smp_affinity ; done
218 -> 00001
219 -> 003ff
220 -> 003ff
221 -> 003ff
222 -> 003ff

Notice also that in the output of fw ctl affinity -l -r -v -a, eth1 seems to use the same IRQ with eth12.

While, in reality:

Click to Expand

# grep "eth1-" /proc/interrupts | awk ' { print $1, $23 } '
258: eth1-0
259: eth1-txrx-1
260: eth1-tx-2
261: eth1-tx-3
262: eth1-tx-4

# grep "eth12-" /proc/interrupts | awk ' { print $1, $23 } '
263: eth12-0
264: eth12-txrx-1
265: eth12-tx-2
266: eth12-tx-3
267: eth12-tx-4

So, my question is... does it look the same in other installations using tg3 driver in r80.40 / is it expected behaviour or should I look for something else?

Thanks!

John     

`

[G_W_Albrecht]

This looks similar sk171526: Output of 'fw ctl affinity' does not show FWK affinity to CPU core after upgrade of R80.30...

The issue you have should be fixed in R80.40 sk166356: 'fw ctl affinity' and "sim affinity" commands showing wrong data

[jkougoulos]

thanks for the pointers!

sk166356 looks similar but a bit different case. Now that I look closer, I believe I can emulate the way that fw ctl affinity matches the interfaces to interrupts (maybe it is the "last resort" method):

# grep eth1 /proc/interrupts | tail -1 | awk ' { print $1 } '
267:
# grep eth12 /proc/interrupts | tail -1 | awk ' { print $1 } '
267:

for sk171526, I believe this installation started with a clean r80.40. Also this relates to FWK affinity, I don't see a problem there:

# fw ctl affinity -l -v
Interface eth0 (irq 222): CPU 0 1 2 3 4 5 6 7 8 9
Interface eth1 (irq 267): CPU 1
Interface eth2 (irq 242): CPU 1
Interface eth3 (irq 247): CPU 1
Interface eth4 (irq 237): CPU 0
Interface eth5 (irq 227): CPU 0 1 2 3 4 5 6 7 8 9
Interface eth6 (irq 217): CPU 1
Interface eth7 (irq 257): CPU 1
Interface eth11 (irq 212): CPU 1
Interface eth12 (irq 267): CPU 1
Interface eth13 (irq 232): CPU 1
Interface eth14 (irq 252): CPU 1
VS_0: CPU 2 3 4 5 6 7
VS_0 fwk: CPU 2 3 4 5 6 7
VS_1: CPU 2 3 4 5 6 7
VS_1 fwk: CPU 2 3 4 5 6 7
VS_5: CPU 2 3 4 5 6 7
VS_5 fwk: CPU 2 3 4 5 6 7
VS_8: CPU 2 3 4 5 6 7
VS_8 fwk: CPU 2 3 4 5 6 7
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.
Interface eth8: has multi queue enabled
Interface eth9: has multi queue enabled

# fw ctl multik stat
ID | Active  | CPU    | Connections | Peak    
----------------------------------------------
 0 | Yes     | 2-7    |          26 |      212
 1 | Yes     | 2-7    |          20 |      212
 2 | Yes     | 2-7    |          23 |      212
 3 | Yes     | 2-7    |          24 |      213
 4 | Yes     | 2-7    |          26 |      212
 5 | Yes     | 2-7    |          24 |      213

[Wolfgang]

@jkougoulos one more thing to observe.... disable your unlicensed cores in the BIOS of your open server hardware. There are some known strange problems if more cores active then licensed.

[jkougoulos]

@Wolfgangthanks for this hint. I saw somewhere that tg3 may enable more channels depending on number of cores, but if I want to reduce the cores to 8 in Bios, probably this would mean to switch to 2x4 cores - moving the load between the 2 processors which might have some performance impact.

I will start with ethtool tuning first and if it plays any role and see if I need to resort to reducing the number of cores. I guess I could also set the affinity manually using commands to /proc/irq/xxx/smp_affinity