Bond interface reports RX-DRP discards whilst bond members show zero counters

[Kaspars_Zibarts]

Here's a puzzle of today (this is R80.40 T118 btw)

why would bond interface show increasing RX-DRP counter whilst it's members are showing zero?

Can't really find any settings nor statistics for bond interfaces.

[Timothy_Hall]

Hmm that is strange, probably some kind of cosmetic bug with the netstat command display.  What do these commands show:

ethtool -i eth2-01

ifconfig bond1

ifconfig eth2-01

ifconfig eth2-02

ethtool -S bond1 (may not work)

ethtool -S eth2-01

ethtool -S eth2-02

ip -s -s link show bond1

ip -s -s link show eth2-01

ip -s -s link show eth2-02

[Kaspars_Zibarts]

Apologies, I should have mentioned that I checked those but nothing extraordinary was there. I excluded zero counters from ethtool output to make it shorter

[Expert@vsx1-int:0]# ethtool -i eth2-01
driver: i40e
version: 2.10.19.82
firmware-version: 6.80 0x80004486 0.0.0
expansion-rom-version:
bus-info: 0000:af:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ifconfig bond1
bond1 Link encap:Ethernet HWaddr 00:1C:7F:6C:4C:7C
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:29416121572 errors:4 dropped:295456 overruns:0 frame:2
TX packets:29548776168 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:20173506356637 (18.3 TiB) TX bytes:21829745516209 (19.8 TiB)

[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ifconfig eth2-01
eth2-01 Link encap:Ethernet HWaddr 00:1C:7F:6C:4C:7C
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:23069137356 errors:0 dropped:0 overruns:0 frame:0
TX packets:15877672064 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15771964654259 (14.3 TiB) TX bytes:11768525707703 (10.7 TiB)

[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ifconfig eth2-02
eth2-02 Link encap:Ethernet HWaddr 00:1C:7F:6C:4C:7C
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:6346991897 errors:0 dropped:0 overruns:0 frame:0
TX packets:13671125111 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4401552187331 (4.0 TiB) TX bytes:10061235936840 (9.1 TiB)

[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ethtool -S bond1 | grep -v ': 0'
no stats available
[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ethtool -S eth2-01 | grep -v ': 0'
NIC statistics:
rx_packets: 23069161949
tx_packets: 15877684543
rx_bytes: 15771984688585
tx_bytes: 11768534741630
rx_unicast: 23061309593
tx_unicast: 15876543641
rx_multicast: 7319920
tx_multicast: 5668
rx_broadcast: 532439
tx_broadcast: 1135238
tx-0.packets: 69645317
tx-0.bytes: 29907647201
rx-0.packets: 1069960527
rx-0.bytes: 874477050253
tx-1.packets: 1626080740
tx-1.bytes: 1373528810199
rx-1.packets: 1271940260
rx-1.bytes: 1120772959316
tx-2.packets: 63815521
tx-2.bytes: 33409570517
rx-2.packets: 921826468
rx-2.bytes: 567743560298
tx-3.packets: 898554030
tx-3.bytes: 513696910731
rx-3.packets: 852519476
rx-3.bytes: 512592358291
tx-4.packets: 80377350
tx-4.bytes: 40271287468
rx-4.packets: 849102287
rx-4.bytes: 538344890844
tx-5.packets: 1036340067
tx-5.bytes: 677595274517
rx-5.packets: 943372393
rx-5.bytes: 577364278484
tx-6.packets: 78321622
tx-6.bytes: 33348986724
rx-6.packets: 1207707794
rx-6.bytes: 754549791890
tx-7.packets: 1285660999
tx-7.bytes: 1075983684188
rx-7.packets: 1209242374
rx-7.bytes: 1013698479643
tx-8.packets: 129636029
tx-8.bytes: 125232434790
rx-8.packets: 725079755
rx-8.bytes: 489943791697
tx-9.packets: 1167908513
tx-9.bytes: 714773737485
rx-9.packets: 795750614
rx-9.bytes: 440245198563
tx-10.packets: 74525208
tx-10.bytes: 30050948616
rx-10.packets: 1037844587
rx-10.bytes: 741778954712
tx-11.packets: 996518907
tx-11.bytes: 732808530557
rx-11.packets: 843485191
rx-11.bytes: 517778661250
tx-12.packets: 73593135
tx-12.bytes: 21481486099
rx-12.packets: 989603117
rx-12.bytes: 769009005154
tx-13.packets: 923347704
tx-13.bytes: 549072980470
rx-13.packets: 1132842136
rx-13.bytes: 609055092242
tx-14.packets: 56755003
tx-14.bytes: 20193514295
rx-14.packets: 1028274546
rx-14.bytes: 632924791868
tx-15.packets: 2452735112
tx-15.bytes: 2114799189985
rx-15.packets: 1002730894
rx-15.bytes: 675573092615
tx-16.packets: 78741045
tx-16.bytes: 33078289178
rx-16.packets: 738533594
rx-16.bytes: 471922136730
tx-17.packets: 1183879366
tx-17.bytes: 946297611396
rx-17.packets: 1022209424
rx-17.bytes: 820541736721
tx-18.packets: 71541980
tx-18.bytes: 32995402328
rx-18.packets: 796245551
rx-18.bytes: 507555339290
tx-19.packets: 1143583624
tx-19.bytes: 931331054517
rx-19.packets: 1033829502
rx-19.bytes: 763043068633
tx-20.packets: 78245563
tx-20.bytes: 35268447301
rx-20.packets: 919644722
rx-20.bytes: 655776437082
tx-21.packets: 995975397
tx-21.bytes: 702326130245
rx-21.packets: 923786796
rx-21.bytes: 651691418737
tx-22.packets: 140595612
tx-22.bytes: 30639530054
rx-22.packets: 865820149
rx-22.bytes: 529881355024
tx-23.packets: 1171306699
tx-23.bytes: 970443282769
rx-23.packets: 887809795
rx-23.bytes: 535721240656
port.rx_bytes: 15949118891369
port.tx_bytes: 11900103641299
port.rx_unicast: 23061309584
port.tx_unicast: 15876543632
port.rx_multicast: 7319920
port.tx_multicast: 5668
port.rx_broadcast: 532439
port.tx_broadcast: 1135238
port.mac_remote_faults: 70
port.rx_csum_bad: 4233
port.rx_size_64: 1847834292
port.rx_size_127: 4065098589
port.rx_size_255: 5247196051
port.rx_size_511: 2048540459
port.rx_size_1023: 954719109
port.rx_size_1522: 8905773449
port.tx_size_64: 2181188747
port.tx_size_127: 1957991767
port.tx_size_255: 2942355461
port.tx_size_511: 1357694794
port.tx_size_1023: 708684794
port.tx_size_1522: 6729768983
port.fdir_flush_cnt: 1
port.fdir_sb_status: 1
[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ethtool -S eth2-02 | grep -v ': 0'
NIC statistics:
rx_packets: 6346993613
tx_packets: 13671131293
rx_bytes: 4401553038516
tx_bytes: 10061239790483
rx_unicast: 6346008669
tx_unicast: 13671124943
rx_multicast: 568552
tx_multicast: 5669
rx_broadcast: 416355
tx_broadcast: 689
tx-0.packets: 1116996178
tx-0.bytes: 895176525391
rx-0.packets: 244306443
rx-0.bytes: 139692523037
tx-1.packets: 60960091
tx-1.bytes: 20217382157
rx-1.packets: 271443463
rx-1.bytes: 98248851402
tx-2.packets: 902313362
tx-2.bytes: 494143311402
rx-2.packets: 168612586
rx-2.bytes: 81121701295
tx-3.packets: 50692914
tx-3.bytes: 14340398604
rx-3.packets: 161845422
rx-3.bytes: 90477008593
tx-4.packets: 865320314
tx-4.bytes: 528076132809
rx-4.packets: 272919353
rx-4.bytes: 147883973863
tx-5.packets: 62469509
tx-5.bytes: 24051225626
rx-5.packets: 171605312
rx-5.bytes: 87108972954
tx-6.packets: 965313946
tx-6.bytes: 587418597925
rx-6.packets: 171484391
rx-6.bytes: 92599983826
tx-7.packets: 80689135
tx-7.bytes: 33091098889
rx-7.packets: 195692037
rx-7.bytes: 89861308259
tx-8.packets: 885204372
tx-8.bytes: 633293145624
rx-8.packets: 180833281
rx-8.bytes: 98458325610
tx-9.packets: 59500620
tx-9.bytes: 25315394990
rx-9.packets: 206979934
rx-9.bytes: 101600183874
tx-10.packets: 1398323949
tx-10.bytes: 1326361380184
rx-10.packets: 400838484
rx-10.bytes: 416094991883
tx-11.packets: 53585409
tx-11.bytes: 18397012709
rx-11.packets: 321007995
rx-11.bytes: 133222811853
tx-12.packets: 1221437166
tx-12.bytes: 973525209824
rx-12.packets: 281862977
rx-12.bytes: 180552883716
tx-13.packets: 48845368
tx-13.bytes: 25821323951
rx-13.packets: 211227612
rx-13.bytes: 153729875602
tx-14.packets: 1017865480
tx-14.bytes: 718744354331
rx-14.packets: 423014980
rx-14.bytes: 293490829009
tx-15.packets: 69183084
tx-15.bytes: 35529078454
rx-15.packets: 169972232
rx-15.bytes: 96831130206
tx-16.packets: 875914534
tx-16.bytes: 556116090238
rx-16.packets: 191099666
rx-16.bytes: 79610389854
tx-17.packets: 69490697
tx-17.bytes: 44304262760
rx-17.packets: 345425522
rx-17.bytes: 265529683872
tx-18.packets: 1041160302
tx-18.bytes: 811173404300
rx-18.packets: 181115178
rx-18.bytes: 103732282435
tx-19.packets: 49333748
tx-19.bytes: 17730423041
rx-19.packets: 255560753
rx-19.bytes: 154183523184
tx-20.packets: 1449928297
tx-20.bytes: 1253573836800
rx-20.packets: 555200962
rx-20.bytes: 635750596545
tx-21.packets: 79018888
tx-21.bytes: 30059260963
rx-21.packets: 284317142
rx-21.bytes: 193989807486
tx-22.packets: 1199559371
tx-22.bytes: 980055919519
rx-22.packets: 258999950
rx-22.bytes: 220709929510
tx-23.packets: 48024561
tx-23.bytes: 14725020212
rx-23.packets: 421627938
rx-23.bytes: 447071470648
port.rx_bytes: 4451336366529
port.tx_bytes: 10175016453434
port.rx_unicast: 6346008666
port.tx_unicast: 13671124936
port.rx_multicast: 568552
port.tx_multicast: 5669
port.rx_broadcast: 416355
port.tx_broadcast: 689
port.rx_csum_bad: 2231
port.rx_size_64: 247255048
port.rx_size_127: 1575640159
port.rx_size_255: 1003323506
port.rx_size_511: 646318607
port.rx_size_1023: 458253463
port.rx_size_1522: 2416202793
port.tx_size_64: 2105869365
port.tx_size_127: 1845424615
port.tx_size_255: 2060688092
port.tx_size_511: 1309411277
port.tx_size_1023: 543770391
port.tx_size_1522: 5805967559
port.fdir_flush_cnt: 1
port.fdir_sb_status: 1
[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ip -s -s link show bond1
62: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000
link/ether 00:1c:7f:6c:4c:7c brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
20173540296218 29416165439 4 295456 0 7890487
RX errors: length crc frame fifo missed
0 2 0 0 0
TX: bytes packets errors dropped carrier collsns
21829775627886 29548817603 0 0 0 0
TX errors: aborted fifo window heartbeat transns
0 0 0 0 4
[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ip -s -s link show eth2-01
22: eth2-01: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc noqueue master bond1 state UP mode DEFAULT qlen 1000
link/ether 00:1c:7f:6c:4c:7c brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
15771987324166 23069165213 0 0 0 7319920
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
11768535910078 15877686329 0 0 0 0
TX errors: aborted fifo window heartbeat transns
0 0 0 0 8
[Expert@vsx1-int:0]#
[Expert@vsx1-int:0]# ip -s -s link show eth2-02
23: eth2-02: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc noqueue master bond1 state UP mode DEFAULT qlen 1000
link/ether 00:1c:7f:6c:4c:7c brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped overrun mcast
4401597521699 6347085310 0 0 0 568552
RX errors: length crc frame fifo missed
0 0 0 0 0
TX: bytes packets errors dropped carrier collsns
10061415700852 13671435740 0 0 0 0
TX errors: aborted fifo window heartbeat transns
0 0 0 0 8

[Timothy_Hall]

Those constantly-incrementing RX-DRPs are almost certainly caused by reception of frames with an unknown EtherType, but normally those counter hits are charged against the underlying physical interfaces, not the bond interface.  This issue was discussed in the third edition of my book pages 208-210, and the specific counter behavior you are seeing might be unique to the i40e driver and/or how it interacts with Multi-Queue. 

Make a note of the number of RX-DRPs on bond1 then either start a tcpdump on the bond1 interface or explicitly set it to promiscuous mode with the ifconfig bond1 promisc command; if that isn't allowed you will need to start tcpdumps or set promisc on the underlying physical interfaces to achieve the same effect.  Do the RX-DRPs stop incrementing as long as promiscuous mode is active?  If so it is just unknown EtherTypes arriving and nothing really to worry about; this RX-DRP effect became much more common with the Gaia 3.10 kernel update.

[Kaspars_Zibarts · Accepted solution]

Thanks Tim for detailed explanation! I eventually managed to "filter out" the offender frames as RX-DRP counter was increasing on standby platform too so considerably less traffic to look at. It was IPv6 packets and we don't have IPv6 enabled 🙂 

Thanks for help anyways guys!

vsx2-int:0> show ipv6-state
ipv6 is disabled

[genisis__]

Strange, on our installation we are not seeing that behaviour (See below) we are also running R80.40 with T118)