Hello,
I am new in Checkpoint firewalls (well I configured a couple of rules 20 years ago in Solaris based one, but this does not count) and I have inherited a VSX HA cluster running on HP DL380 Gen10 Openserver with 3 quad tg3 based (broadcom) 1G NICs and a couple of dual Intel 10G NICs. The box has 2x10 cores, the first 8 of them are licensed.
I have read a lot of documents and skXXXXX in the last days, many forum posts and have done several experiments. I still have some issues with setting up the affinity of the interfaces properly.
Long story short, it looks like each tg3 interface is serviced by 5 IRQs
eg:
Click to Expand (Highlight to read) # grep "eth0-" /proc/interrupts | awk ' { print $1, $23 } '
# grep "eth0-" /proc/interrupts | awk ' { print $1, $23 } '218: eth0-0219: eth0-txrx-1220: eth0-tx-2221: eth0-tx-3222: eth0-tx-4
Now, when I setup the affinity of the interface eth0 to CPU 0, this is what I get:
Click to Expand (Highlight to read) # fw ctl affinity -s -i eth0 0
# fw ctl affinity -s -i eth0 0Interface eth0: CPU 0 - set successfullyMulti-queue affinity was not changed. For More info, see sk113834.# fw ctl affinity -l -r -v -aCPU 0: eth0 (irq 222) eth4 (irq 237) eth5 (irq 227)CPU 1: eth0 (irq 222) eth1 (irq 267) eth2 (irq 242) eth3 (irq 247) eth5 (irq 227) eth6 (irq 217) eth7 (irq 257) eth11 (irq 212) eth12 (irq 267) eth13 (irq 232) eth14 (irq 252)CPU 2: eth0 (irq 222) eth5 (irq 227)CPU 3: eth0 (irq 222) eth5 (irq 227)CPU 4: eth0 (irq 222) eth5 (irq 227)CPU 5: eth0 (irq 222) eth5 (irq 227)CPU 6: eth0 (irq 222) eth5 (irq 227)CPU 7: eth0 (irq 222) eth5 (irq 227)CPU 8: eth0 (irq 222) eth5 (irq 227)CPU 9: eth0 (irq 222) eth5 (irq 227)CPU 10:CPU 11:CPU 12:CPU 13:CPU 14:CPU 15:CPU 16:CPU 17:CPU 18:CPU 19:All:The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.Interface eth8: has multi queue enabledInterface eth9: has multi queue enabled# fw ctl affinity -lInterface eth0: CPU 0 1 2 3 4 5 6 7 8 9Interface eth1: CPU 1Interface eth2: CPU 1Interface eth3: CPU 1Interface eth4: CPU 0Interface eth5: CPU 0 1 2 3 4 5 6 7 8 9Interface eth6: CPU 1Interface eth7: CPU 1Interface eth11: CPU 1Interface eth12: CPU 1Interface eth13: CPU 1Interface eth14: CPU 1VS_0: CPU 2 3 4 5 6 7VS_0 fwk: CPU 2 3 4 5 6 7VS_1: CPU 2 3 4 5 6 7VS_1 fwk: CPU 2 3 4 5 6 7VS_5: CPU 2 3 4 5 6 7VS_5 fwk: CPU 2 3 4 5 6 7VS_8: CPU 2 3 4 5 6 7VS_8 fwk: CPU 2 3 4 5 6 7The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.Interface eth8: has multi queue enabledInterface eth9: has multi queue enabled
Now if I go a bit deeper:
Click to Expand (Highlight to read) # for i in `grep "eth0-" /proc/interrupts | awk -F: ' { print $1 } ' | sed 's/://' `; do echo -n "$i -> " ; cat /proc/irq/$i/smp_affinity ; done
# for i in `grep "eth0-" /proc/interrupts | awk -F: ' { print $1 } ' | sed 's/://' `; do echo -n "$i -> " ; cat /proc/irq/$i/smp_affinity ; done218 -> 00001219 -> 003ff220 -> 003ff221 -> 003ff222 -> 003ff
Notice also that in the output of fw ctl affinity -l -r -v -a, eth1 seems to use the same IRQ with eth12.
While, in reality:
Click to Expand (Highlight to read) # grep "eth1-" /proc/interrupts | awk ' { print $1, $23 } '
# grep "eth1-" /proc/interrupts | awk ' { print $1, $23 } '258: eth1-0259: eth1-txrx-1260: eth1-tx-2261: eth1-tx-3262: eth1-tx-4# grep "eth12-" /proc/interrupts | awk ' { print $1, $23 } '263: eth12-0264: eth12-txrx-1265: eth12-tx-2266: eth12-tx-3267: eth12-tx-4
So, my question is... does it look the same in other installations using tg3 driver in r80.40 / is it expected behaviour or should I look for something else?
Thanks!
John
`
