ISP Redundancy Load Sharing Issue w/ 2 BGP Peers (inactive route from one peer)

[cmale]

I have two 9100s running in an H/A cluster and two BGP peers (ISP-A & ISP-B).

I am established with both ISPs, but only accepting routes from one. I am receiving the full routing table from one, ISP-A. I am receiving only the default from ISP-B. 

I see one route (default) coming in from ISP-B, but it is not being accepted (show route all).

PeerID           AS           Routes  ActRts  State             InUpds  OutUpds  Uptime
		    	        1       0       Established       2       1        02:25:33
	          	     1033184  1033184  Established       241826  1        02:31:30

I have Inbound Route filters for both peers set to accept routes. I do not see a BGP route from ISP-B (show route bgp). It is only showing routes from ISP-A. Show route all shows the ISP-B route as 'inactive'.

S               0.0.0.0/0           via yy.yy.yy.yy, eth5, cost 0, age 9304
                                    via xx.xx.xx.xx, eth2
B            i  0.0.0.0/0           via yy.yy.yy.yy, eth5, cost None, age 8903 

I have routemaps set up for BGP for both peers to accept routes as well. Inbound Route Filters are configured in Gaia.

 (*) = Routemap not configured

Routemaps for eBGP AS yyyyy

    Default

        Import: None

        Export: None

    Peer "yy.yy.yy.yy"

        Import: None

        Export: None

Routemaps for eBGP AS xxxxx

    Default

        Import: None

        Export: None

    Peer "xx.xx.xx.xx"

        Import: None

        Export: None

Routemaps for iBGP AS x.xxxx

    Default

        Import: None

        Export: None

Routemap : "bgp"

    Id : ----- [permit]

            Match Conditions:

                Protocol : bgp

            Actions:

I tested by shutting down my standby appliance and on the active appliance, unplugging the physical link for ISP-A. It is at this point that we lose internet when I am left with only ISP-B.

Anyone work through something similar? I feel like I'm missing something obvious.

[simonemantovani]

During the test, what was the status of the BGP peers? 

Normally you should have both BGP peers established, and for all the routes received via BGP, you should see active routes for the peer A, and inative routes on peer B.

Could you also check what routes do you receive from peer B using the command: show bgp peer <peerB IP address> adj-rib-in

[cmale]

ISP-B was Established and ISP-A was Idle during the test. 

Sure, here is the output:

---------------------- EBGP Peer yy.yy.yy.yy (AS yyyy) -----------------------

IPv4 Route                                   Nexthop

0.0.0.0/0                                    yy.yy.yy.yy

[simonemantovani]

Ok good, were you able to check where the request to Internet where redirected? On which WAN interface? In addition, how do you configured NAT for outbound traffic? Automatic?

[cmale]

No, I did not check that. NAT is automatic.

[Duane_Toler]

You don't need "match protocol bgp"; it's redundant.  You do need, at minimum, "set routemap FOO id 10 allow" tho and add that to an import-routemap for the peer (or peer group, which is often best).  You also can't use routemap *and* inbound-route-filter at the same time.  Use routemaps; that's your better choice and better control.

You don't get ECMP by default unless you turn it on, but you may or may not want that.  Without ECMP, you won't get the second default route installed by BGP. 

BGP has a hold time (default 180 seconds) before it will drop routes from a dead peer (assuming the gateway's own local interface link didn't go down).  If you want faster detection of a dead peer, then you will want to use BFD (set bgp external remote-as 65515 peer 192.0.2.1 ip-reachability-detection on) but the peer has to enable it as well.  If you enable BFD, you also want to enable the C-bit to do control-plane checking.  If you can't, or don't want to, do BFD, you can also enable ping instead if the peer allows you to ping it (set bgp external remote-as 65515 peer 192.0.2.1 ping on).

Your "show route" (the FIB) is only going to show the "valid-and-best" routes selected by the BGP process; "show route all" is just the RIB, which is just a collection of possible routes from all routing protocols (static, BGP, EIGRP, RIP, OSPF, etc.)  RIB is nice-to-have, but FIB is best.

Make sure your peer is peering to your cluster VIP *only* and not to each gateway member individually.  If they are peering to each gateway, you absolutely will have weird issues during any failover event.

Hope some of this helps!

[Chris_Atkinson]

Seeing the raw BGP config at this point would probably be helpful.

In addition to NAT as others have mentioned, what's the source address and how is that being advertised / reached?