Connectivity down between a 9100 gateway and a cisco 4500 module

[Josh28]

Hello,

I’m facing an issue with a new 9100 cluster, trying to connect it to an old 4500 (specificaly a WS-X4306-GB card) but all ports remain in the state « down (notconnect) » (4 ports in total, on both member of the cluster so I’m rejecting a connection issue). Below some outputs :

Firewall2> show asset network

Number of line cards: 1

Line card 1 model: CPAC-8-1/10F-D

Line card 1 type: 8 ports 1/10GbE Fiber Rev 1.0

Firewall2> show interface eth1-02

state on

mac-addr xx:xx:xx:xx:xx:xx

type ethernet

link-state link down

mtu 1500

auto-negotiation off

speed N/A

ipv6-autoconfig Not configured

monitor-mode Not configured

duplex N/A

link-speed 1000M/full

comments

ipv4-address Not Configured

ipv6-address Not Configured

ipv6-local-link-address Not Configured

Firewall2> show interface eth1-02 xcvr_detail 

eth1-02 SFP is present

Product Type: 10G Base-SR

Vendor name: FINISAR CORP.

Vendor PN: FTLX8574D3BCL

Vendor rev: A

Vendor SN: xx

Laser wavelength: 850nm

Link Length for SMF,km: 0km

Link Length for SMF: 0m

Link Length for 50um: 80m

Link Length for 62.5um: 30m

Link Length for Copper: 0m

Link Length for OM3: 300m

No tx fault, No rx loss

Router2#show interfaces Gi2/6

GigabitEthernet2/6 is down, line protocol is down (notconnect)

  Hardware is Gigabit Ethernet Port, address is xxxx.xxxx.xxxx (bia xxxx.xxxx.xxxx)

  Description: Firewall 2

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, link type is force-up, media type is 1000BaseSX

For me, transceivers seem good on both ends. On router side, there is nothing much I can configure on the port expect of a « speed nonegotiate » which doesn’t change the behavior. On Checkpoint side, I’ve forced the speed and duplex to match the router’s but without a change either :

Router2#sh run int Gi2/6                     

interface Firewall2

 description NS_RESA_U142018_FWVTECH

 speed nonegotiate

end

set interface eth1-02 link-speed 1000M/full

set interface eth1-02 state on

set interface eth1-02 auto-negotiation off

Do you know if there is some known incompatibility between the new quantum firewall and old cisco modules ?

Thank you.

[Chris_Atkinson]

To clarify what brand / SKU of SFP is populated in the ports, does HCP complain about them?

Also per sk92755 not all of them support multirate capabilities.

[Josh28]

Hi, Thanks for your answer. HCP doesn't complain about the SFP on both member of the cluster:

| System/Hardware/Transceivers Support
|
+-----------------------------------------------------------------------------------------------------------------
-------------------+
| Result: SUCCESS
|
|
|
| Description: This test checks that all installed transceivers are supported
|
|
|
| Summary:All transceivers are approved

[Chris_Atkinson]

Great the remaining aspect is the speed / multirate issue and if the SFP supports it (refer sk92755).

[Josh28]

Hi,

Thank you all for your feedback, I’m waiting to get the proper 1 GbE SFP for the gateways to see if it fixes the issue.

Meanwhile, to give you more context, I’m trying to upgrade our links because I’ve noticed some TX-DRP on one of the interfaces of our bonding:

[Expert@Firewall1:0]# netstat -ni

Kernel Interface table

Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg

Mgmt       1500   0 16778174      0      0      0 78205393      0      0      0 BMRU

bond1      1500   0 5697476654      0      0      0 5549913947      0  21368      0 BMmRU

eth1       1500   0 3480924870      0      0      0 2919267926      0      0      0 BMsRU

eth2       1500   0 2216551784      0      0      0 2630646021      0  21368      0 BMsRU

lo        65536   0 39523309      0      0      0 39523309      0      0      0 ALMNORU

[Expert@Firewall1:0]# ifconfig eth2

eth2        Link encap:Ethernet  HWaddr 00:1C:7F:C9:26:D5 

            UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

            RX packets:2216576196 errors:0 dropped:0 overruns:0 frame:0

            TX packets:2630672317 errors:0 dropped:21368 overruns:0 carrier:0

            collisions:0 txqueuelen:2048

            RX bytes:1499929022177 (1.3 TiB)  TX bytes:1996154942442 (1.8 TiB)

What’s troubling is the output of the ethtool below:

[Expert@Firewall1:0]# ethtool -S eth2 | grep 21368

     ife_oqdrops: 21368

Anyone know what “ife_oqdrops” could be referring to ?
Thank you.

Edit: I'll add that the gateway is a new 9100 appliance running UPPAK, replacing a 5600 appliance which didn't had any of those drops, with the same traffic going through them. 

[the_rock]

Im sure @Timothy_Hall would give you way better explanation than I can, but to me, that sounds like its attempting to send/transmit way more data than what buffer would allow.

[Timothy_Hall]

Seeing errors on the TX side in the output of netstat -ni is a strong indicator that UPPAK is active, since TX-side errors were extremely rare in KPPAK mode.  To my understanding, that counter indicates that packets were pushed into the TX ring buffer faster than they could be transferred to the NIC, and some were lost.  However, the eth2 interface is part of a bond. If it leads to a transit VLAN, ensure your Transmit Hash Policy for that bond is L3+4, not the default L2 XOR, as the qdrops may have been caused by improper balancing of traffic between the bond interfaces.  Please see my Be your Own TAC: Part Deux presentation for more information about this issue.

[Josh28]

Thank you for your feedback, and your presentation, which is helpful.

We did change the Hash Policy on the gateway after noticing a load-balancing issue with the bond and the TX drops, but the other end (an old 4500 router) doesn’t support a similar load-balancing method, so it doesn’t help. Thus, now we’ll configure the bond to use Gigabit interfaces on the router to see if it help with the drops.

[Timothy_Hall]

Setting the Transmit Hash Policy to L3+4 should still help with your TX errors.  As long as you are not seeing RX problems, the Transmit Hash Policy on the 4500 does not need to match, although you may see RX traffic imbalances on the firewall interfaces.

[the_rock]

If you do ifconfig and show interfaces from clish, does it show as up in both places?