VSXHA missing routes on External Virtual Router context

[Milos_Jovovic]

Hello,

We are facing with the following issue in our VSXHA (active/standby) deployment (2x23500 GW's, Gaia R80.30):

VSX topology is as follows:

        Outside world

                  I

----bond100_3000-----

External_vRouter (VS1)

--------wrpj128--------

                  I

--------wrp128--------

Virtual Firewall/system (VS2)

-----bond200_299-----

                  I

Inside networks (services)

VS0 has all setup and working smoothly (routes, interfaces, policy,...).

VS1 (vRouter) is direct contact with outside networks/Internet via bond100.3000 interface. It has it's own routing defined. 

Virtual System 2 (firewall under this external vrouter) has connection to inside networks via bond200.XXX interfaces. And connection with vrouter via wrp link wrp128 (we put here private ip 172.11.1.1 for PtP virtual link).

The problem is that VS1 (virtual router) goes to down state on one VSX cluster member/GW when we put default route to VS2 (virtual firewall), pointing to vrouter object. When we delete this default route on VS2 topology, down state goes to standby as should on VS1.

Other problem is that when VS1 is initialy configured all routes defined are correctly pushed to both GW's (VS1 context), and states of VS1 are stable active/standby.

But after that, when we define create VS2 and define inter-connection with VS1 and default route to VS1 (vrouter), VS1 state on one GW goes to down. 

After rebooting no matter which GW (GW1 or GW2), that rebooted GW comes up with deleted static routes for defined for VS1 context (virtual router). 

We have tried everything we know, deleting VS1, VS2 and recreate both again, but all time we are facing with exactly the same issue with route deletion after rebooting GW's, and this wrp link issue.

WRP link on VS2 is defined as Leads to Internet (in topology) but it does not create default route automatically on VS2. We need to create default route that is over vrouter because of that, but then VS1 state goes to down on one GW.

We investigated reason for this Down state and it is like active device interface check PROBLEM for this wrp128 interface.

On DOWN VS1 GW (inboud DOWN, outbound DOWN) on Active VS1 GW (inboundUP, outbound DOWN).

We tried to switch CCP to broadcast without success, to change kernel parameter for routing/topology cync between cluster members to 1, no success.

Please advice here,

[Kaspars_Zibarts]

do you mind sharing topology screenshots on both VS, I'm not 100% sure if I understand wrp link config between VSes

[Kaspars_Zibarts]

BTW did you try using unnumbered interfaces? You don't need to specify IPs for WRP links, check VSX admin guide

[Milos_Jovovic]

Hello Kaspars,

I can not share here the topology as it is with customer's real IP's.

I tried with unnumbered wrp interface on VS2 (virtual firewall) but it is the same, the VS1 (virtual router goes to Down state and static routes are not present on VS1 context on GW's after reboot). All the same as with numbered interfaces.

It is crazy that after I remove default route on VS2, VS1 status settle down (active/standby) and wise versa (down/active) if I add droute on VS2. 

And it is really crazy that I tested, prior to creating VS2 (virtual firewall), i have only vrouter (VS1). And if I configure static routes, during VS1 creation all routes are on GW's. But if I reboot GW's prior to creating VS2, those VS1 routes are deleted frou GW's routing table. 🙂

Missing routes are not those that VS1 (vrouter gets from VS2 where are defined to Propagate to adjacent Virtual System devices. Those missing routes are all connected with those manually defined on MGMT console (VS1 object/topology).

[Milos_Jovovic]

One more thing to add: all three (VS0, VS1, VS2) security policies are set to accept -  ANY - ANY for temporary for testing purposes.

[Kaspars_Zibarts]

Have you tried not propagating routes between VSes?

It's still very confusing the actual setup. Would be hard to help without screenshots. See if you can work around it 🙂

[Kaspars_Zibarts]

I just have a feeling that you are possibly propagating default route both ways vs1 <> vs2 and that in turn would create infinite loop..

[Milos_Jovovic]

I even do not add default route on VS1 (do not add it here) as i want BGP later on VS1 (default route will be received by bgp peer).

[mdjmcnally]

I think a topology diagram would be great as struggling to follow 

I think what saying is that 

1.) Built VSX Cluster in HA

2.) Defined VS1 which is a Virtual Router - Connected to physical interface bond100.3000

3.) Defined a Second VS2 which is a Virtual System - Connected to Physcial Interfaces bond200.xxx

4.) Define an interface on the VS2 leading to VS1 using the 172.11.1.1.   This is the wrp128 link presumably with the automatic ip assignment for the cluster members

5.) Configure Routes on the VS1 for networks off the bond200.xxx interfaces on VS2 and say next hop is Virtual System VS2.

All is good in terms of cphaprob at this point

6.) Configure Route on VS2 for Default Route pointing too VS1 as the Next Hop.

At this point you are saying that VS1 on the Standby Cluster Member goes to a down state and loses routes.

Do you have the Automatic Topology Calculation on VS1 and VS2 enabled.    I find with VSX is easiest to do this.

On the Interfaces then for the Cluster Members then do you have it set to Automatic member IP's.

Not 100% certain by this line

WRP link on VS2 is defined as Leads to Internet (in topology) but it does not create default route automatically on VS2

Only options will be 

Regular

Virtual Router

Virtual Switch

It sounds as though set to Regular and then under the Address Spoofing for the Interface set to External.

What I would expect is that would be set to Virtual Router and then VS1 selected.

You then configure a Default Route in the Routes and make the Next Hop VS1

Topology Calculation would then update the Topology for that interface to be External

That I believe will fix your issue

[Magnus-Holmberg]

I would really try to change the design to avoid the PAIN of the VR.
We have had some horrible experience with it over the years and its something that limits your use of VSLS and other nice VSX features.

[Milos_Jovovic]

1.) Built VSX Cluster in HA

Right

2.) Defined VS1 which is a Virtual Router - Connected to physical interface bond100.3000

Right

3.) Defined a Second VS2 which is a Virtual System - Connected to Physcial Interfaces bond200.xxx

Right

4.) Define an interface on the VS2 leading to VS1 using the 172.11.1.1.   This is the wrp128 link presumably with the automatic ip assignment for the cluster members

Right

5.) Configure Routes on the VS1 for networks off the bond200.xxx interfaces on VS2 and say next hop is Virtual System VS2.

I have done it this way now, because I removed on VS2 (propagate routes to other VS's). And VS1 routes on GW's are reflecting this manually defined routes on MGMT server's VS1 object.

But, in the very next step i add two more static routes on VS1 which points to external next hop IP (or default route - the same), and  they are not populated on VS1 routing table on GW's. This is silly.

All is good in terms of cphaprob at this point

Still Right.

6.) Configure Route on VS2 for Default Route pointing too VS1 as the Next Hop.]

Yes. And on VS2 routing table always reflects changes on MGMT server VS2 object in topology. 

At this point you are saying that VS1 on the Standby Cluster Member goes to a down state and loses routes.

Right. VS1 state on standby member goes from Standby to Down after adding default route on VS2. On active GW state stays active.

Do you have the Automatic Topology Calculation on VS1 and VS2 enabled.    I find with VSX is easiest to do this.

Yes, Calculate topology automatically option is turned on on both VS's.

On the Interfaces then for the Cluster Members then do you have it set to Automatic member IP's.

Not 100% certain by this line

WRP link on VS2 is defined as Leads to Internet (in topology) but it does not create default route automatically on VS2

Only options will be 

Regular

Virtual Router

Virtual Switch

It sounds as though set to Regular and then under the Address Spoofing for the Interface set to External.

I set always wrp link on VS2 as - leads to Virtual Router (VS1). Can not choose external because i previously turned on automatic topology calculation. But it is the same problem with automatic topology calculation tuned off.

What I would expect is that would be set to Virtual Router and then VS1 selected.

Right.

You then configure a Default Route in the Routes and make the Next Hop VS1

Right.

Topology Calculation would then update the Topology for that interface to be External

That I believe will fix your issue

The Issue is still there. 🙂 (state down of VS1 on one GW, and routing table of VS1 on MGMT is not updated on GW's). 

If i remove WRP link from VS2, only then routing table on VS1 updates on GW's, BUT without any manually added route. And default route is removed from VS2. The only route is for my directly connected network (automatically created with my external bond100.3000 interface). On VS1 all manually created routes for bond200.XXX networks are deleted and all manually created routes for external networks (behined next hop) are in place on mgmt server, but non of them are on GW's routing table -only one route for interface outside network.

And after wrp link removal, VS1 state normalizes (active/standby). 🙂

Ather that, If I add again wrp link on VS2, only 172.11.1.1/32 is directly connected, wrpj128 adds to VS1 routing table no other static VS1 routes.

Then if I add static routes on VS1 which points to VS2 and only these routes are added to VS1 routing table on GW's.

Please people advice here 🙂

[Milos_Jovovic]

VS1 (External_vRouter)

External interface - bond100.3000 (public IP)

Internal Interface (wrp interface leading to VS2 - Virtual Firewall/System, automatically added to VS1 when wrp interface created on VS2)

We want on VS1 to have two static routes /32 pointing to our two external routers (BGP VS1 peers later), from which VS1 will receive Default route and to which will VS1 send wanted/needed static routes). This routes if added are not applied to VS1 routing table on GW’s.

On VS1 we can manually add static routes for internal networks (under VS2 bond200.XXX interfaces), or to go with an option of sharing these routes on VS2 topology setup - which are defined on VS2 (VS1 will this way receive this routes to internal networks from VS2 automatically). This routes (added on both variants) are applied to VS1 routing table on GW’s.

VS2 (virtual system/firewall)

bond200.XXX is Internal interface to internal resources/services.

VS2 have manually added static routes to Internal resources/subnets over internal next hop’s. This communication works as a charm all time (during all problems).

wrp128 is virtual PtP link between VS1 and VS2 (problems with routes and VS1 state are present with numbered as well with unnumbered option chosen on wrp interface).

When we add default route to VS2, which points to our VS1 automatically VS1 goes to Down state on one GW.

If we delete wrp link from VS2 topology, just then VS1 routing table makes some change (but again the only present route in routing table is automatically created for bond100.3000 interface). All other static routes in VS1 topology are not in routing table.

[Milos_Jovovic]

Hello All,

We solved all problems here with changing VSLS to HA mode 🙂 

We thought it was already HA mode as it allowed us to create and configure vRouter. 

Thank You all for Your time and contribution.