Performance Limitations of Virtual Switches in a VSX Environment

[HeikoAnkenbrand]

I’ve noticed that in a VSX environment the virtual switches don’t seem to achieve very high packet throughput.
In practice, I can’t get more than around 4–5 Gbps over a wrp interface. When I connect the same setup using physical switches + 100Gbps transceiver and enable multiqueueing, I can reach about 80 Gbps on a 100 Gbps interface .

The setup I tested consists of two scenarios with two 29000 appliances for VSX LS:
In the first one, Virtual System 1 is connected to Virtual System 2 through a virtual switch within the VSX environment.
In the second scenario, Virtual System 1 and Virtual System 2 are connected through a physical switch. 

This allows me to directly compare the performance of traffic flows when using virtual switching versus physical switching.

So my questions are:

• What are the limitations of virtual switches in VSX regarding throughput?

• Are virtual switches vs. wrp (Warp Link) capable of using multiple cores for packet forwarding?

[Gera_Dorfman]

We’re not aware of any virtual switch performance limitations in KPPACK. However, I’ll double-check with the team and get back to you. To clarify again - we are aware of the issue with UPPACK and working on a fix. 

[jan-j]

Hi. Any news on double-checking with the teams if this is also an issue on KPPAK? Thx.

[PhoneBoy]

The (apparent) lack of multi-core may be part of the issue here.
Not sure if KPPAK has the same issue or not, but given we will be UPPAK only from R82.10, I expect we'll prioritize fixing it there.

[genisis__]

Maybe I'm thinking too deep about this, but would a VSW not utilise SND Cores to push traffic through, if that is the case, would dynamic balance or hyperflow then kick-in.??  

[PhoneBoy]

As I recall, SND leverages RSS (Receive Side Scaling, not to be confused with Really Simple Syndication).
That happens at the NIC level, as far as I know.
Not sure how that translates to VSes, exactly.

Hyperflow is only relevant to traffic passed in the Medium Path, which I assume is not the case for a VSW.

[genisis__]

Thanks.

[Andreas_Zentsch]

Hi Gera,

I have a design-related question regarding VSX topologies.

I have seen many customer environments where the VSX infrastructure is effectively built as a mesh. In these scenarios, a single Virtual Switch connects three or more firewalls through WRP interfaces within the same subnet. Physically, each firewall only has one connection towards the switching infrastructure, but logically the VSX forwarding calculations allow traffic to flow directly between all participating firewalls across the shared Virtual Switch segment.

What I currently struggle with is understanding whether this design actually provides a real advantage compared to a cleaner star-based topology.

I could not find any explicit recommendation in the official administration guides. However, reading between the lines, it feels as if a star design is the preferred or at least cleaner approach.

Theoretically, asynchronous routing should not occur because VSX itself calculates and controls the forwarding path. On the other hand, the Virtual Switch still processes the traffic for all connected firewalls. In my example, one Virtual Switch effectively handles the traffic exchange between three different firewalls simultaneously.

This leads me to another question: if the environment were redesigned using multiple Virtual Switches instead of one large shared segment, would the actual traffic load on the physical switching infrastructure change at all?

From my understanding, the amount of traffic traversing the physical switches would remain nearly identical, because the same traffic still has to pass through the same underlying physical links. The main difference would only be the logical segmentation and distribution across multiple Virtual Switch instances rather than a real reduction of traffic on the switch fabric itself.

Do you have any official guidance, best practices, or operational experience regarding whether a mesh-style VSX topology is considered a recommended design, or whether a star topology should generally be preferred?

Andreas

[emmap]

It's more about what works for your network. You must use a Virtual Switch if you want to share an interface (access port or sub-interface on a trunk) between Virtual Systems. If you want to have two VSs sharing a VLAN and avoid using a VSw, you need to end up adding the same VLAN to two trunks (or two access ports to the same VLAN). Typically we find that the networking crew would prefer we didn't use more interfaces than we need, so we deploy VSw's. From a routing side of things, there's not really any difference either way - if you want to traverse two VSs that are linked by that VLAN, you need routes on each to do that. If there's a VSw there, there are some advantages - you can propagate routes between VSs by ticking that option (in legacy VSX anyway) and we can avoid some double handling in SXL if the packet doesn't leave the box between each VS. 

[Henrik_Noerr1]

Hey Andreas, just to state the obvious. Creating a VSW and connecting Virtual Systems does not change the inter routing between Virtual Systems at all.

This only happens if you mark interfaces, or routes to be propagated in the topology screen.

If marked the subnet route will be pushed via the VSW to all connected Virtual Systems.

I am unsure if this is fixed, but propagating a large amount of routes, it resulted in a large boot time penalty for our clusters taking more than an hour to load virtual systems. This resulted in us converting to routing the subnets to the physical router, then back into the VSX cluster taking whatever performance hit this resulted in, but it fixed the boot issues.

This was on r81.20

/Henrik  

edit: I could also have read the post further down by emmap - stating much of the same thing 🙂

[Timothy_Hall]

Hi Heiko,

The PMTR number I found does not match the one in Gera's very informative post, but I came across this in the R82.10 known limitations and thought of this thread immediately, even though it is talking about connection rates and not throughput.  Might be worth a shot to see if it helps performance in your virtual switches:

PMTR-120253: In VSX mode, when running performance tests at higher CPS (Connections per second) across different sources and destinations, the Security Gateway will experience drops because that source-based routing is enabled in VSX mode by default.  To resolve: Disable source-based routing to reduce the number of cache entries created and deleted during CPS test and improves the performance. Run: fw ctl set -f int cphwd_enable_ecmp 0 -a

[genisis__]

Would be interesting to see if this works, but its not something I would want to try in a live setup, especially with a large dynamic routing table.
Maybe Checkpoint can  confirm if its safe to do this?