weird issue with 40Gbit Module and new 29100 Plus appliances

[AleLovaz82]

Hi

we have 6 spare (for now there isnt' any cluster configuration ) 29100 plus appliances ( 81.20 Take 84 ) ,each one is equipped with 1 of this module

2 Port 40GBase QSFP+ / 100GBase QSFP28 interface card for 9000/19000/29000 appliances - CPAC-2-40/100F-D.

Both the port of EACH gateway are connected to a cisco ACI LEAF where all the ports are configured in the same way.

What is my problem?
Only the port2 of EACH modulo goes up.
Is not a Cisco ACI issue or a QSFP+ issue or a fiber issue because we tried to

change port on cisco ACI moving ,and in every port  ...module port2 is working,module port1 doesn't.

• move all the QSFP and fiber from port2 to port1 , on port2 everything is working fine,in port1 we see only "link down".
  Remember that this issue occurs on SIX spare 29100 with SIX different module connected to different leaf on ACI and we tried to change/swap anything
  
  The only difference I can see from CP pov is this one :
  
  [Expert@:0]# ethtool --show-fec eth4-01
  FEC parameters for eth4-01:
  Configured FEC encodings: Auto
  Active FEC encoding: Auto
  [Expert@0]# ethtool --show-fec eth4-02
  FEC parameters for eth4-02:
  Configured FEC encodings: Auto
  Active FEC encoding: Off
  
  the "Active FEC encoding" it's indipendent and cannot be changed with " ethtool --set-fec eth4-01 encoding 'parameter' "
  Even if I change Fec Encoding to none ,or RB or Baser the "Active FEC encoding" remain Auto in port eth4-01 and Off in port eth4-02
  
  Any idea?
  
  
  
  

[Bob_Zimmerman]

The forward error correction is negotiated. "Configured FEC encodings" is what you can control. "Active FEC encoding" tells you what value was negotiated, and "Auto" just means it hasn't negotiated a value. It's like the difference in lspci output between LnkCap (Link Capabilities) and LnkSta (Link State; a card which is capable of 8 GT/s and 16x width could be used in a slot supporting only 2.5 GT/s and 4x).

What does 'ethtool -m' show on each interface? Do they both recognize their transceivers? Do both transceivers show received light?

[AleLovaz82]

 ethtool -m eth4-01
Identifier : 0x11 (QSFP28)
holy s**t ,it's the 100gbit trasnceiver. The people in datacenter told us that they used QSFP+ in both the ports.
This should be the issue, thx for the hint , I never head the -m flag on ethtool !
tomorrow we'll see ,someone will change them

[Bob_Zimmerman]

Was that the problem? Seems unlikely six different boxes would all have the wrong transceivers inserted in the same slot and the right transceivers inserted in the same slot.

[AleLovaz82]

Hi Bob, 
wrong SFP in each port 2 of each 29100, and also a cabling issue -.-'