BGP routes are missing after ClusterXL failover

[Wei_Soon_Heng]

Hi All,


I am facing a strange issue whereby the BGP session is established successfully with fw02 after failover but are unable to get advertised BGP Routes from SDWAN VeloCloud. Both CheckPoint firewalls are enabled with graceful restart options.

BGP session and routes are working good when fw01 is the active member.

Below is my topology:
Cisco Nexus (AS X)<---> CheckPoint Cluster(AS X) <----> SDWAN VeloCloud (AS Y)

After searching the /var/log/routed.log , There are some lines showing that CP GAIA OS is not supporting some capabilites of BGP,

Please refer to log below:
Sep 6 11:05:30.940081 bgp_get_open(3073): peer 10.25.x.x+21144 (proto) has provided 4 Byte AS 6xxxx
Sep 6 11:05:30.940081 bgp_get_open: peer 10.25.x.x+21144 (proto) received unrecognized capability 69. Ignoring capability 69
Sep 6 11:05:30.940081 bgp_get_open: peer 10.25.x.x+21144 (proto) received unrecognized capability 73. Ignoring capability 73
Sep 6 11:05:30.940081 bgp_pp_recv: Receiving OPEN from peer 10.25.x.x +15501 [eBGP AS 6xxxx] in ESTABLISHED state, entering Graceful Restart Helper mode
Sep 6 11:05:30.940081 bgp_event: peer 10.25.x.x+15501 [eBGP AS 6xxxx] old state Established event RecvOpen new state Idle
Sep 6 11:05:30.940081 bgp_graceful_restart_close_stale_connection: Peer 10.25.x.x+15501 [eBGP AS 6xxxx] does not support non-stop forwarding for any AFI/SAFI, remove all routes from him
CHANGE X.X.X.X /31 gw 10.25.x.x BGP
pref 170/- metric /100 bond2.43 <Ext|Delete|Gateway> as 6xxxx
CHANGE X.X.X.X /24 gw 10.25.x.x BGP
pref 170/- metric /100 bond2.43 <Ext|Delete|Gateway> as 6xxxx
CHANGE X.X.X.X /32 gw 10.25.x.x BGP

It is resolved by disabled the graceful restart feature in fw02 only. So I having fw01 (enabled graceful restart) and fw02(disabled graceful restart).

Hope someone enlighten on why it is still working at fw01 even this fw is enabled with graceful restart options?

FW version is R80.40 with jhf take 102.

Thanks

[vinceneil666]

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/116189-problemsolution-tec...

I am not sure if this will actually fix it, but I had a similar issue sone time back that got resolved setting the non-capabilities on the Cisco end..  Also, I think that there has been some fix for this in R81.10 

What CP version are you rinning ? 

[Wei_Soon_Heng]

FW is running version R80.40 with jhf take 102.
Unfortunately, the peer is not Cisco, it is SDWAN VeloCloud device.

[Chris_Atkinson]

Is it a clean install or in-place upgrade from an older version?

I would check the route-maps / filter lists are uniform on both but also verify FIBMGR traffic per sk109401.

[the_rock]

I had seen this before and firewall reboot had to be done to fix it.

[Wei_Soon_Heng]

problem still exists after reboot of problematic secondary fw

[the_rock]

I would open TAC case...cant find much on those errors at all. If reboot did not clear it, there could be a bigger issue here.

[Andre_K]

Make sure your 'import-routemap' configuration matches on both firewall members, it seems like your BGP peering is up but you're not accepting any BGP routes due to a missing routemap.

[the_rock]

Another thing that came to my mind was maybe do a quick comparison of BGP on both members...just go to clish and run show bgp, hit tab and it will give you all the options to run the command. Its possible something might be missing on the fw2 member. Just a thought...