Routing not working towards VTI

[Darren_Fine]

We have a HA cluster that requires a new AWS VPN connection. The tunnel comes up and we can ping across the VTI's no issues.😁

However when adding a route to the remote VTI interface the route does not show up on the gaia routing table.😫

So for example the tunnel interfaces are

ClusterIP -  vpnt11 = 169.254.192.2 and we can ping 169.254.192.1

ClusterIP - vpnt12 = 169.254.8.250 and we can ping 169.254.8.149

If we add a static route

#set static-route 10.0.1.5/32 nexthop gateway address 169.254.8.149 o

and look at output of...

#show route destination 10.0.1.5

the output shows the nexthop as following the default Internet route.  (---not the VTI---)

I thought about perhaps overlapping encryption domains clashing so added a temporary routes to different addresses (non private) and they have the same result- they dont show in the routing table towards the VTI.

I then remade the vpn with BGP enabled at AWS.

BGP would not establish with an error - "unable to find interfaces to reach this peer"  (even though I can ping the peer)

I enabled multihop and then BGP established. (even though I can ping the peer as a connected interface😂)

If I look at the routes I am learning from the BGP relationship it appears that the next hop is the default gw and not the 169.254.8.149 / 169.254.192.1 (ie not the bgp peer IP's).

What can cause this ?

How can I resolve this ?

(have remade the vpn twice - also recreated it on the AWS side since you cant swap from static to bgp without deleting- very strange- again vpn is up - bgp is allowed and seen decrypted on the vpn correctly - can ping inside the vti)

Look forward to any thoughts.

GW is R80.20 

thanks

[PhoneBoy]

Is there a particular reason you are running R80.20 instead of a more recent release?
The fact the route isn’t adding seems like something the TAC should investigate, but I suspect this might be fixed in a later release.

[Vladimir]

You are mentioning cluster HA, please confirm that it is ClusterXL and not VRRP.

Do you have a rule permitting BGP to the cluster object and to the cluster members from AWS peers?

Are your cluster members' VTI's IPs adjasent to the cluster vIP for VTIs?

Multihop should stay disabled, even though you seem to be getting "better" results with it.

[Darren_Fine]

Hi Vladimir,

Yes this is clusterxl.

The BGP is logged and decrypted successfully within the VPN in the logs.

So is the ping to the internal VTI remote IP encrypted successful ly and working 100%.

So they are adjacent and established.

We have opened a TAC case it looks like a Gaia VPN tunnel interfaces weirdness/issue ...

Hoping TAC can explain..otherwise guess will try the hotfix in case it magically fixes this.

(There is another older aws tunnel working 100% on these box es...don't know why another one would be an issue)

The connected routes look wrong and I believe this is the issue...(if I do a #show route direct ..the routes for the existing VPN looks correct but not for the new tunnel interfaces....even though I have deleted and r made the tunnel interfaces twice ....their direct connected routes do not look correct)

[Vladimir]

Probably stupid question, but I have to ask: since you have mention existing operating tunnels to AWS on the same cluster, can you confirm that the routes you are expecting from the new one are not overlapping the ones already received from the old?

[Darren_Fine]

Hi Vladimir,

So the old one is using static routing. I did first implement static routing on this tunnel but when I could not add a route down the tunnel I decided to try BGP. I did try send other arbitrary static route sdown the VTI  - this also does not appear in the routing table towards the VTI.  (even legal IP's dont work) 

In further testing the BGP routes are seen and can be added but then they get added to the Default GW instead of the VTI interface 🙂

So this is still an issue - hoping TAC can advise.

[Vladimir]

Can you confirm that the router ID is identical on both cluster members and that it is one of the cluster's vIPs (doesn't matter which one, typically picked one that is unlikely to change)?

Not sure it would have any bearing on the static routes, but it may have an impact on BGP.

[Darren_Fine · Accepted solution]

Hi Guys,

So TAC wasnt really sure to be honest.

Anyhow we decided to try and upgrade the backup to the latest hotfix and switch over to it.

This we did and everything worked straight away.

Once on the secondary I was able to get permission to reboot the primary as a test.

Once the primary came back I could see the connected routes were correct and I could see the static route pointing to the VTI internal remote IP (the bgp peer) was successfully populated.

So for some reason a reboot "fixed" it .

Strange I know...