Maestro RX Ring Buffer Configuration

[Danny]

To mitigate RX-DROPs on a Maestro Security Group I wanted to increase the RX ring buffer on the backplane interfaces.

Background:

After rerouting a high PPS 100G traffic flow through a Maestro Security Group I observed increasing RX-DROPs on the active chassis backplane interfaces.

Investigation showed the drops were reported as rx_out_of_buffer while SND and worker cores remained largely idle. Based on the observed behavior, increasing the RX ring buffer was recommended as a mitigation for potential microburst-induced queue overflows.

Related SKs

• sk183094
• sk156132
• R82 Scalable Platforms AdminGuide

Show current settings:

• Gaia Clish (excerpt for SG member 1)
  

  [Global] FWSG-s01-01> show interface BPEth0 rx-ringsize
  1_01:
  Receive buffer ring size: 256
  Maximum receive buffer ring size: 4096

• Expert mode
  

  # grep ringsize /config/active
  
  # cat /proc/net/bonding/BPEth0 | grep eths
  Slave Interface: ethsBP1-01
  Slave Interface: ethsBP1-02
  Slave Interface: ethsBP1-03
  Slave Interface: ethsBP1-04
  
  # ethtool -g ethsBP1-01
  Ring parameters for ethsBP1-01:
  Pre-set maximums:
  RX:             8160
  Current hardware settings:
  RX:             1024

Increasing the RX ringsize:

• Expert Mode (requires a maintenance window as some SG members will reboot)
  

  # g_all -a "dbset interface:ethsBP1-01:rx-ringsize 4096"
  # g_all -a "dbset interface:ethsBP1-02:rx-ringsize 4096"
  # g_all -a "dbset interface:ethsBP1-03:rx-ringsize 4096"
  # g_all -a "dbset interface:ethsBP1-04:rx-ringsize 4096"
  # g_all -a "dbset :save"​

Verify the increased RX ringsize:

• Expert mode
  

  # grep ringsize /config/active
  interface:ethsBP1-01:rx-ringsize 4096
  interface:ethsBP1-02:rx-ringsize 4096
  interface:ethsBP1-03:rx-ringsize 4096
  interface:ethsBP1-04:rx-ringsize 4096​
  
  # ethtool -g ethsBP1-01
  Ring parameters for ethsBP1-01:
  Pre-set maximums:
  RX:             8160
  Current hardware settings:
  RX:             4096

• Gaia Clish (excerpt for SG member 1)
  

  [Global] FWSG-s01-01> show configuration interface
  set interface ethsBP1-01 rx-ringsize 4096
  set interface ethsBP1-02 rx-ringsize 4096
  set interface ethsBP1-03 rx-ringsize 4096
  set interface ethsBP1-04 rx-ringsize 4096
  
  [Global] FWSG-s01-01> show interface BPEth0 rx-ringsize
  1_01:
  Receive buffer ring size:256
  Maximum receive buffer ring size:4096

• Persistence validation
  After rebooting the Security Group the configuration remained present in grep ringsize /config/active
ethtool -g continued to report RX = 4096 on the slave interfaces.
  This was observed without adding any commands to rc.local as documented in sk183094.

Documentation gaps and validation questions:

• Questions
  1. Why does Gaia Clish continue to report 256?
  2. Is ethtool -g on the slave interfaces the authoritative validation method?
  3. Is modifying rc.local still required on R82 Maestro?
  4. Why does Gaia automatically generate set interface ethsBP* rx-ringsize entries after a successful dbset operation, and are these intended to be valid Gaia configuration commands or merely a representation of the underlying database configuration? I'm asking because these commands are rejected when executed manually.
  5. When increasing the RX ring buffer to mitigate RX-DROPs, should the TX ring buffer always be increased as well (as shown in sk183094) or only when transmit-side issues are observed?
  6. Why are four slave interfaces shown behind BPEth0 when only one backplane connection is physically connected, and is this expected Maestro behavior?
  7. Is configuring ethsBP* now the officially recommended method?
  8. Can the SKs be updated with complete validation examples?

---
@Lari_Luoma @Anatoly @Tom_Kendrick @Laszlo_Csosza @Jochen_Hoechner @Tom_Hartig @Timothy_Hall