Question: Max number of Virtual Systems per VSX

[FedericoMeiners]

Hello everyone,

Hope you doing well, I'm designing an architecture to support more than 50 virtual systems with the available resource from a customer.

Deployment Distributed deployment with R80.30 (SMS and gateways) - 2 appliances to create an active/passive VSX Cluster, each VS will have their own separate interfaces.

One of the things that I'm facing going through all the documentation of VSX to verify the deployment feasibility is the max number of virtual systems per VSX. The main limitation that I found is detailed in sk99121 due to the internal communication network.

By default the ICN (192.168.196.0/22) supports a max number of 62 virtual systems with a maximum of 64 interfaces per VS, things get tricky if you have a number VS with more than 64 interfaces:

The limit of VS descends to 30 and then 15 if you need between 128 and 265 interfaces per VS. As far as I can see, this limitation is inherent to how VSX works.

I've been through the admin guide and many SKs but couldn't find a way to overpass this limitation. I though about using IPv6 only in the internal communication network but there's no enough information for this.

Are my assumptions correct here? Maybe there is something that I'm missing. How do you deal with a large scale VSX deployment in this case? Maybe the only way is to create different VSX gateways/clusters.

Thank you in advance!

[Maarten_Sjouw]

I think you left out the most important line that picture being:
The maximal number of interfaces supported by a VSX Gateway / VSX Cluster Member is limited to 4096 interfaces (including physical, VLAN and Warp interfaces).
I have been running into a different limitation a while back, the memory use of each VS has been growing from R80.10 towards R80.30 by about 100-150 percent. Memory allocated per VS starts around 800MB and I have seen it grow up to 1GB per VS.

A configuration with this many interfaces is not very common. Are you able to share what kind of solution you are looking at?

[PhoneBoy]

I know under optimal conditions, the absolute max you can do is 250.
I don't think I've ever seen anywhere near that in reality.
Also, pretty sure the ICN must be IPv4.

Meanwhile, I believe at least some of these limits can/will be lifted due in part to the new Linux kernel.
More details about your proposed architecture would be helpful.

[FedericoMeiners]

Maarten, PhoneBoy,

Thank you for your inputs.

I've been doing more research with our local SE and it seems that the limitation of Max interfaces vs Max VS can't be bypassed, only way is to use Security Groups on Chasis or Maestro solutions.

To add background, the customer currently has a pair of ASA 5500 firewalls which have around 50+ VS, it seems that Cisco does not have a limitation on max interfaces vs max VS. 

What we did is to split the VS with more than 64 interfaces so we have a higher cap of max VS per VSX cluster.

@Maarten_Sjouw Thanks for the heads up, We managed to handle RAM/CPU by doing a really intense IPS tunning, for the moment we are aiming to use only Firewall + IPS with various optimizations. So far so good.

Thanks!

[Lari_Luoma]

This sk explains the relation between the number of VSs and interfaces per VS.
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

The theoretical max. number of 250 VSs is only for a gateway that is not dependent on the funny IP-network, mainly we talk about a chassis here.

[HeikoAnkenbrand · Accepted solution]

Hi @FedericoMeiners 

The maximal number of interfaces supported by a VSX Gateway / VSX Cluster Member is limited to 4096 interfaces.

By default, Virtual System can be connected to a maximum of 64 different Virtual Switches / Virtual Routers.
To be able up to configure more than 64 interfaces, you should use a bigger network mask for internal communication network. Therefore more VSs are possible here.

I see more problems in this theoretical discussion with CPU cores (CoreXL) and multi queueing and and and.

Please show me the appliance or open server with 4096 network interfaces or more than 64 cores. Okay, a 64K has a few more cores. But the discussion here is very theoretical😀

@Lari_Luoma 
With 250 VSs the tuning becomes interesting.😀

[Lari_Luoma]

You guys are correct. 250 Virtual Systems is a theoretical maximum for scalable platforms (44K/64K). However, it's not the recommendation and as Heiko mentioned, optimizing it could be a nightmare.. I have seen some environments with close to 100 VSs, but the real max. number I can recommend is about half of that. 50 VSs on one cluster is big environment that still works fine.

Of course it also depends on the load on the VSs. You can easily have 50 small VSs, but if each one of them is pushing one million connections you will reach the limit much earlier. 🙂 Here we come to one of my favorite discussions with customers... VSX was designed to have several small(ish) virtual firewalls. If you really need to have very big environments, consider using a physical devices for them. I have seen some customers running only one big VS per cluster. What's the point when you could use it as a regular gateway? 

If you have to ask "what's the maximum number of this or that" you most likely already have an architectural problem. 🙂

[Magnus-Holmberg]

Honestly i would like to say that 25-30 is more the limit.
we have a few clusters with 50 and they are very slow to make routing changes on in the VS.
Performance itself is no issue, but adding/changing interfaces routes takes several minutes instead of seconds.
During a maintenance window for a small change like changing default gw for a VS this is painfull.

We are currently running on R80.30 3.10.

Regards,
Magnus

[FedericoMeiners]

I'll gladly share with you the lessons learned after I finish, I found that there are many "theorical" areas regarding VSX architecture that I'm looking to research 🙂

@HeikoAnkenbrand So far MultiQ is not an issue since it's configured in physical interfaces, most of our interfaces are VLANs therefore the MultiQ is only configured on 2 SFP links. ClusterXL it's a little bit trickier in large scale deployments of VSX but so far we are managing it.

Again, thank you all for your advises and thoughts, you are the best

Federico Meiners