VSNext inter-vs communication

[melcu]

Hi Expert,

I have a setup with 3 gateways running R82 ElasticXL in VSNext.

vs0 - management, bla bla

vs1 - servers

vs2 - "other servers" + wifi + users + whatever

Requirements:

VS2 "wifi" should comunicate with vs1's servers

vs1 servers need to communicate with vs2 'other servers'.

What I am not understanding in VSNext is:

 - if you create one Virtual Switch but do not assign an external (physical) interface can you use it by "connec to Vritual Swtich" inside each VS and just put a  /30 in between ?  Then  static routes   from vs1 to x.x.x.x/24 from vs2  via interconnect

from vs2 return static routes /24 via vs1 interconnect.

Will this work ?

[Bob_Zimmerman]

That's exactly how it would work, yes.

[Global] DallasticXL-s01-01:0> add vsnext virtual-[ESC][ESC]
add vsnext virtual-gateway interfaces VALUE id VALUE one-time-password VALUE instances VALUE instances6 VALUE wait-for-task VALUE
add vsnext virtual-link virtual-gateway VALUE virtual-switch VALUE wrp-id VALUE
add vsnext virtual-switch name VALUE id VALUE wait-for-task VALUE

When building a new switch VS, you just need to specify a name and an ID (which can be "auto" to let the system assign a VSID).

Once the switch exists, you add warp links by specifying the non-switch VSID, the switch VSID, and the warp ID (which can also be "auto"). Remember to add this to the gateway's topology in the management and to push policy for antispoofing to work correctly.

From there, it's like any other transit interface. I know older VSX wouldn't let you use /31 blocks, and I'm not sure if VSNext resolved that limitation. A /30 should be fine, just wasteful. You can either add static routes or use dynamic routing (e.g, with a route-map to redistribute connected routes so as you add new interfaces on either VS, the other will just pick up the routes).

I'm not sure if switch VSs run any form of STP, so be careful with your topology.

[Wolfgang]

@Bob_Zimmerman  and @melcu I believe you need to assign a physical or VLAN interface for such a connection. Iˋm not really sure maybe something changed with VSnext but with VSX you need this interface. With VSX and ClusterXL you can have a configuration Running one VS1 on clusternodeA and Running VS2 on clusternodeB. If you want connectivity between both VSs you need an external switch and assigned interfaces and too the virtual switch. All packets are flow through the external switch.

I believe with ElasticXL there is too a need with such a configuration.Maybee @emmap or @Lari_Luoma can comment this.

[melcu]

Thanks but what will be the reason to call it "virtual" if you need to route between external :))
I will spin up 2 gateways in EXL and try to do a /30 between VS 1 and VS2 and ping between.

If you're asking me is a stupid to use an external interface 😞 It's not like when you have VSLS were  VS1 sits in Netherlands and VS2 is in Japan.

[Bob_Zimmerman]

That definitely wasn't a requirement on classic VSX.

I was once involved in a datacenter connection project with constantly changing requirements. We ordered new firewalls based on the "final requirements", but the project team changed them again before the boxes were even delivered. Now they wanted something with a given address in one DC to be able to talk to something with the same address in the other DC, so we needed two NAT points. Complex political situation resulted in us being told we had to deliver without any additional purchases. The final deployment was VS0 handling traffic on one side, a switch for VS1, then VS2 handling traffic for the other side. This is possible using any firewall license (switch VSs don't consume a license slot, and every license comes with VS0 plus one VS). You just need to use vsx_util vsls to tweak VSLS distribution to stick all the VSs on one member, otherwise a lot of traffic can end up on the sync link.

VSNext has an advantage here, though: there's no more Standby state. All the VSs are active on all the members.

[Bob_Zimmerman]

Just tested, and it works. Also, /31 networks evidently work now.

[Global] DallasticXL-s01-01:0> add vsnext virtual-switch name TestSwitch id auto wait-for-task true 
Please wait while the Virtual System is being created
Task ID        Status              Age       Action    
001-002-81443  100% (Completed)    15s       Create Virtual System - Finished

Notes:
* Virtual link to the Management Switch will be created automatically

[Global] DallasticXL-s01-01:0> add vsnext virtual-link virtual-gateway 0 virtual-switch 2 wrp-id auto 

[Global] DallasticXL-s01-01:0> add vsnext virtual-link virtual-gateway 1 virtual-switch 2 wrp-id auto 

[Global] DallasticXL-s01-01:0> set interface wrp1 ipv4-address 192.168.200.0 mask-length 31 
1_01:
success
...

[Global] DallasticXL-s01-01:0> set virtual-system 1
Context is set to vsid 1

[Global] DallasticXL-s01-01:1> set interface wrp65 ipv4-address 192.168.200.1 mask-length 31
1_01:
success
...

[Global] DallasticXL-s01-01:1> exit

[Expert@DallasticXL-s01-01:0]# ifconfig wrp1 | grep -Po "(?<=HWaddr )[0-9A-F:]+"
00:12:C1:10:10:FC

[Expert@DallasticXL-s01-01:0]# vsenv 1
Context is set to Virtual Gateway DallasticVS1 (ID 1).

[Expert@DallasticXL-s01-01:1]# ifconfig wrp65 | grep -Po "(?<=HWaddr )[0-9A-F:]+"
00:12:C1:10:18:FC

[Expert@DallasticXL-s01-01:1]# vsenv 2
Context is set to Virtual Switch TestSwitch (ID 2).

[Expert@DallasticXL-s01-01:2]# ifconfig -a | egrep "^[^ ]" | cut -d' ' -f1
br2
erspan0
gretap0
gre0
lo
wrpj1
wrpj65

[Expert@DallasticXL-s01-01:2]# vsenv 0
Context is set to Virtual Gateway DallasticXL (ID 0).

[Expert@DallasticXL-s01-01:0]# m 1_2
Moving to member 1_2

[Expert@DallasticXL-s01-02:0]# ifconfig wrp1 | grep -Po "(?<=HWaddr )[0-9A-F:]+"
00:12:C1:20:10:FC

[Expert@DallasticXL-s01-02:0]# vsenv 1
Context is set to Virtual Gateway DallasticVS1 (ID 1).

[Expert@DallasticXL-s01-02:1]# ifconfig wrp65 | grep -Po "(?<=HWaddr )[0-9A-F:]+"
00:12:C1:20:18:FC

[Expert@DallasticXL-s01-02:1]# arp -n 192.168.200.0
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.200.0            ether   00:12:c1:10:10:fc   C                     wrp65

In the ifconfig output from VS2, you can see there's no external interface. While s01-02 sees s01-01's MAC, packet captures on s01-01 don't show traffic I send at all. Not on Sync, not on wrp1, not on br2.

Meanwhile, I get this in a packet capture on s01-02 VS2 when I ping from s01-02 VS1 wrp65 to VS0 wrp1:

[Expert@DallasticXL-s01-02:0]# vsenv 2
Context is set to Virtual Switch TestSwitch (ID 2).

[Expert@DallasticXL-s01-02:2]# tcpdump -ennvi br2 
tcpdump: listening on br2, link-type EN10MB (Ethernet), capture size 262144 bytes
16:37:15.582504 00:12:c1:20:18:fc > 00:12:c1:10:10:fc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 61132, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.200.1 > 192.168.200.0: ICMP echo request, id 59192, seq 1, length 64
16:37:15.583041 00:12:c1:20:10:fc > 00:12:c1:10:18:fc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 65058, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.200.0 > 192.168.200.1: ICMP echo reply, id 59192, seq 1, length 64
16:37:16.582676 00:12:c1:20:18:fc > 00:12:c1:10:10:fc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 61138, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.200.1 > 192.168.200.0: ICMP echo request, id 59192, seq 2, length 64
16:37:16.583100 00:12:c1:20:10:fc > 00:12:c1:10:18:fc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 65167, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.200.0 > 192.168.200.1: ICMP echo reply, id 59192, seq 2, length 64

Edit to add:

I ran cpview on s01-01 looking at Network > Interfaces > Traffic, then started a flood ping (ping -f 192.168.200.0) from s01-02 VS1. My Sync bond and its member went up from ~160 PPS to ~11k PPS. Clearly this traffic crosses the sync interface, so that interface's throughput is a limit on the potential throughput over the warp.

[this_guy_again]

Wolfgang is correct, per sk183336.

You need a physical switch to bridge the virtual switches inside each physical host.

[melcu]

So what you are telling me is that if I have 2  and I need  VS1 to communicate with VS2 basically I need to connect  ethX from each SGM to an external layer2 swtich  and then route.

What is the difference in connecting ethx  from  SGM1 to ethx to SGM2  and connecting  both ethX  to an external swtich. 
Who wrote the code for this **bleep** ?

Fortigate has inter-vdom communication and guess what it can do 80Gbps (ask me how I know). It's accelerated by the ASIC. 

Junper has lsys/tsys within its belly.

Check Point .. let's buy some SFPs, let's consume some ports because we need VS1 to communicate with VS2.

So stupid design. Really. SO stupid!

[this_guy_again]

Hmm, you might get away with a back-to-back connection, yeah. As long as the virtual switches are connected... Best to be tested.

[emmap]

It doesn't have to be a whole new interface, a sub-interface/VLAN on an existing trunk will work. I believe the reason for this requirement is for Distribution. The packet might be distributed to SGM 1_1 for VS1 but then SGM 1_2 for VS2. We don't want to load up the Sync interface with all the inter-VS traffic here so we do the pivotting over the VSwitch's 'network' interface. If you only have one SGM per site then you don't need this, as long as your cluster is in HA mode not VSLS mode.

[Lari_Luoma]

A VSW needs a physical interface in ElasticXL. I talked with R&D about this one.

[melcu]

What if I use an existent interface (both gateways have 2x 100Gbps bonds connected  to the outside).
What if I create it like another subinterface , let's say  vlan 3096 .

Gateway 1:
bond1.3096   ip: 192.168.1.1/30

Gateway2:

Bond2.3096  ip: 192.168.1.2/30

Then just add statics from vs1 to vs2 via 192.168.1.2 and from vs2 to vs1 via 192.168.1.1.
vlan3096 will be added to that external stack of switches and it will be tagged along with other vlans.

To put additional interfaces on that 19200s costs big $$$$

[this_guy_again]

The link between the two virtual gateways will be WRP interface on each VS leading to the same vSwitch. Then you would assign the IP Addresses as you said.

The vSwitch will need an interface (or sub-interface also works, yes) which leads to the outside world, so that the vSwitches inside each physical node can pass data between them in the scenarios mentioned earlier in the thread.

But yes, sub-interface also works. You create a VLAN in your physical switch for the "inter-VS communication", you create a sub-interface on your VSNext setup assigned to this VLAN, and assign the sub-interface to the vSwitch when creating the vSwitch.

virtual gw1 -- wrp -- vswitch -- sub-interface on physical node 1 -- physical switch -- sub-interface on physical node 2 -- vswitch -- wrp -- virtual gw2

[Bob_Zimmerman]

My testing above shows this is incorrect. Sending the traffic out to physical switches is still a good idea, since without an external link, the traffic hits the Sync bond. Still, an external interface is definitely not a hard requirement.

[Lari_Luoma]

Did you test it with ElasticXL? Did you test the failover as well? I know it works in Maestro for sure. I verified this with R&D, so it's not based on my opinion. We do have features that might work without being fully supported because in some scenarios they might fail.