topic Re: Unable to set MQ on more than 10 cores R80.30 T215 2.16 kernel in Firewall and Security Management

Unable to set MQ on more than 10 cores R80.30 T215 2.16 kernel

Kaspars_Zibarts — Wed, 07 Oct 2020 19:10:00 GMT

I was trying to add two more MQ cores to my 23800 VSX cluster (10>12) but after increasing ixgbe core count and reboot, one of 5 interfaces that had MQ configured before would remain on "Pending ON" state

cpmq set rx_num ixgbe 12

[Expert@vsx:0]# cpmq get -a Active ixgbe interfaces: eth2-01 [On] eth2-02 [Off] eth2-03 [On] eth2-04 [Pending On] eth3-01 [On] eth3-02 [On] Non-Active ixgbe interfaces: eth4-01 [Off] eth4-02 [Off] eth4-03 [Off] eth4-04 [Off]

even it says clearly that ixgbe should support 16 cores in the performance tuning admin guide

But then from command line it shows 10!

[Expert@vsx:0]# cpmq get rx_num ixgbe The rx_num for ixgbe is: 10

Ouch! Anyone knows if I can get more than 10 on 2.16 kernel R80.30? Is it a bug or wrong documentation

Just noticed that there is override option, has anyone tried it?

Re: Unable to set MQ on more than 10 cores R80.30 T215 2.16 kernel

Kaspars_Zibarts — Wed, 07 Oct 2020 20:01:03 GMT

OK, -f did not help so I suspect since this is non HT system with total of 24 cores there must be some calculation in place that limits max cores for ixgbe MQ to 10..

Re: Unable to set MQ on more than 10 cores R80.30 T215 2.16 kernel

Timothy_Hall — Wed, 07 Oct 2020 20:59:52 GMT

The driver-based max limit for ixgbe is 16 queues, but Multi-Queue is partially implemented by the underlying NIC hardware which can have lower limits. As an example the igb driver theoretically supports up to 8 queues, but if the underlying NIC hardware is I211 (onboard interfaces for 3200, 5000, 6500, 15000, 23000) it can only have a maximum of two parallel queues per interface. I suspect this is why you can't go beyond 10 queues in your case and attempting to go higher fails, which leads to the persistent "Pending On" state.

Re: Unable to set MQ on more than 10 cores R80.30 T215 2.16 kernel

PhoneBoy — Thu, 08 Oct 2020 04:23:33 GMT

Believe the older Linux kernel supports a maximum of 5 interfaces with multiqueue.

Re: Unable to set MQ on more than 10 cores R80.30 T215 2.16 kernel

Kaspars_Zibarts — Thu, 08 Oct 2020 04:59:29 GMT

Yap, that's correct, but I wanted to add a bit more horse power to SND on those five as we were suffering from voice quality in teams due to very bursty nature of traffic. CPU (60%) and interface utilisation (25%) all looked good but we still saw 0.18% RX-DRP on some of the interfaces. Increased RX ring buffers to MAX and it helped a little initially. But then as Tim has been suggesting back in March adding more SND cores may help even though they do not report 100% utilisation. So we went from 8 > 10 MQ cores first and that showed massive improvement in RX-DRP. Since I was able to free up more cores in VSX, I wanted to go all the way to 12 but run to this limitation. So next step is 3.10 which was not available back in March for 23800 appliances 🙂

My old reference https://community.checkpoint.com/t5/VSX/RX-DRP-rx-missed-errors-on-VSX-23800-R80-30-with-mltiqueue/td-p/79802