topic Re: Mellanox/NVIDIA ethtool RX ring buffer size request in Firewall and Security Management

Mellanox/NVIDIA ethtool RX ring buffer size request

Timothy_Hall — Wed, 21 Jun 2023 13:41:46 GMT

Can someone who is using the following Mellanox/NVIDIA network cards please run an ethtool -g (interface) from expert mode on them and post the outputs? I curious to see what the default RX ring buffer size is for these interfaces utilizing the mlx drivers. Specifically I'm interested in seeing outputs for these connected line speeds:

25Gbps
40Gbps
100Gbps

For Intel, it looks like the default RX ring buffer size for igb is 256 and ixgbe is 512. Thanks!

Desired output should look like this:

[Expert@A-GW-01:0]# ethtool -g eth0

Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 4096
TX: 4096
Current hardware settings:
RX: 1024
RX Mini: 0
RX Jumbo: 256
TX: 1024

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

Danny — Wed, 21 Jun 2023 14:51:52 GMT

Type: Check Point 16600HS Appliance
Line card: Model: CPAC-2-100/25F Type: 2 ports 100GbE QSFP28

Interface: ethsBP1-01
Identifier : 0x11 (QSFP28)
Transceiver type : 40G Ethernet: 40G Base-CR4
Length (Copper or Active cable) : 1m
Vendor name : Mellanox
Product Type : 100GBASE-CR4-1M
Speed : 100G
Check Point SKU : CPAC-DAC-100G-1M-B

[Expert@CheckMates]# ethtool -g ethsBP1-01
Ring parameters for ethsBP1-01:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 1024

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

Hugo_vd_Kooij — Tue, 12 Mar 2024 17:20:32 GMT

We have cards set to a RX buffer of 512 after installing Jumbo Hotfix Take 41 on R81.20 where it was running Take 29 before this.

Actually it was set to 512 on all interfaces of the QLS450 appliance.

Adn we got loads of RX-DROPS

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

John-Haynes — Tue, 12 Mar 2024 18:54:32 GMT

Ring parameters for eth1-01:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 1024
RX Mini: 0
RX Jumbo: 0
TX: 1024

Model: Check Point 29100

Line card 1 model: CPAC-2-40/100F-D
Line card 1 type: 2 ports 40/100GbE Fiber Rev 1.0

driver: mlx5_core
version: 5.8-2.0.3 (21 Jul 23)
firmware-version: 28.37.1014 (CP_0000000014)

Transceiver type : 100G Ethernet: 100G Base-SR4 or 25GBase-SR

Vendor name : PROLABS
Vendor OUI : 00:00:00
Vendor PN : 13B8NIY4456
Vendor rev : A2

Currently, any setting above 1024 with UPPAK enabled will cause crashes.

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

Timothy_Hall — Wed, 13 Mar 2024 17:15:24 GMT

Thanks, looks like 1024 is more or less the default for 40/100 interfaces.

> Currently, any setting above 1024 with UPPAK enabled will cause crashes.

Really? That is very strange, are you using a Lightspeed appliance? Are we talking a system-wide panic and reboot here or just crashing of the UPPAK SecureXL process?

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

John-Haynes — Wed, 13 Mar 2024 18:34:57 GMT

29100 with connectx-7 cards. USIM coredumps and then it reboots.

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

Timothy_Hall — Wed, 13 Mar 2024 18:43:38 GMT

Yikes, thanks!

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

MattLeGrow — Thu, 14 Mar 2024 20:11:10 GMT

Hi John,

Was traffic passing through the device at the time of the ringsize change?

As a general comment, the Ringsize settings for devices in User Space mode may differ from prior kernel mode defaults, even for the same card - these variations are indeed deliberate.

If not already, I recommend opening a case with Check Point TAC and providing the usim core dumps for analysis, as well as /var/log/usim_x86.elg* and /var/log/messages* for additional context.

Re: Mellanox/NVIDIA ethtool RX ring buffer size request

Timothy_Hall — Wed, 20 Mar 2024 23:02:47 GMT

I couldn't envision how touching a ring buffer setting could ever crash SecureXL, but it looks like UPPAK/USIM mode is digging some tendrils into the Linux networking code in ways it never has before, which as a side effect is impacting use of the standard Linux ethtool command: sk181564: Differences using the 'ethtool' command with SecureXL Kernel Mode (KPPAK) and User Mode (UPPAK) modes