Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
FedericoMeiners
Advisor

Question: Max number of Virtual Systems per VSX

Jump to solution

Hello everyone,

Hope you doing well, I'm designing an architecture to support more than 50 virtual systems with the available resource from a customer.

Deployment Distributed deployment with R80.30 (SMS and gateways) - 2 appliances to create an active/passive VSX Cluster, each VS will have their own separate interfaces.

One of the things that I'm facing going through all the documentation of VSX to verify the deployment feasibility is the max number of virtual systems per VSX. The main limitation that I found is detailed in sk99121 due to the internal communication network.

By default the ICN (192.168.196.0/22) supports a max number of 62 virtual systems with a maximum of 64 interfaces per VS, things get tricky if you have a number VS with more than 64 interfaces:

VS limitation.png

The limit of VS descends to 30 and then 15 if you need between 128 and 265 interfaces per VS. As far as I can see, this limitation is inherent to how VSX works.

I've been through the admin guide and many SKs but couldn't find a way to overpass this limitation. I though about using IPv6 only in the internal communication network but there's no enough information for this.

Are my assumptions correct here? Maybe there is something that I'm missing. How do you deal with a large scale VSX deployment in this case? Maybe the only way is to create different VSX gateways/clusters.

Thank you in advance!

 

____________
https://www.linkedin.com/in/federicomeiners/
0 Kudos
Reply
1 Solution

Accepted Solutions
HeikoAnkenbrand
Champion
Champion

Hi @FedericoMeiners 

The maximal number of interfaces supported by a VSX Gateway / VSX Cluster Member is limited to 4096 interfaces.

By default, Virtual System can be connected to a maximum of 64 different Virtual Switches / Virtual Routers.
To be able up to configure more than 64 interfaces, you should use a bigger network mask for internal communication network. Therefore more VSs are possible here.


I see more problems in this theoretical discussion with CPU cores (CoreXL) and multi queueing and and and.

Please show me the appliance or open server with 4096 network interfaces or more than 64 cores. Okay, a 64K has a few more cores. But the discussion here is very theoretical😀

@Lari_Luoma 
With 250 VSs the tuning becomes interesting.😀

View solution in original post

0 Kudos
Reply
8 Replies
Maarten_Sjouw
Champion
Champion
I think you left out the most important line that picture being:
The maximal number of interfaces supported by a VSX Gateway / VSX Cluster Member is limited to 4096 interfaces (including physical, VLAN and Warp interfaces).
I have been running into a different limitation a while back, the memory use of each VS has been growing from R80.10 towards R80.30 by about 100-150 percent. Memory allocated per VS starts around 800MB and I have seen it grow up to 1GB per VS.

A configuration with this many interfaces is not very common. Are you able to share what kind of solution you are looking at?
Regards, Maarten
PhoneBoy
Admin
Admin
I know under optimal conditions, the absolute max you can do is 250.
I don't think I've ever seen anywhere near that in reality.
Also, pretty sure the ICN must be IPv4.

Meanwhile, I believe at least some of these limits can/will be lifted due in part to the new Linux kernel.
More details about your proposed architecture would be helpful.
0 Kudos
Reply
FedericoMeiners
Advisor

Maarten, PhoneBoy,

Thank you for your inputs.

I've been doing more research with our local SE and it seems that the limitation of Max interfaces vs Max VS can't be bypassed, only way is to use Security Groups on Chasis or Maestro solutions.

To add background, the customer currently has a pair of ASA 5500 firewalls which have around 50+ VS, it seems that Cisco does not have a limitation on max interfaces vs max VS. 

What we did is to split the VS with more than 64 interfaces so we have a higher cap of max VS per VSX cluster.

@Maarten_Sjouw Thanks for the heads up, We managed to handle RAM/CPU by doing a really intense IPS tunning, for the moment we are aiming to use only Firewall + IPS with various optimizations. So far so good.

Thanks!

____________
https://www.linkedin.com/in/federicomeiners/
0 Kudos
Reply
Lari_Luoma
Employee
Employee

This sk explains the relation between the number of VSs and interfaces per VS.
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

The theoretical max. number of 250 VSs is only for a gateway that is not dependent on the funny IP-network, mainly we talk about a chassis here.

 

 

0 Kudos
Reply
HeikoAnkenbrand
Champion
Champion

Hi @FedericoMeiners 

The maximal number of interfaces supported by a VSX Gateway / VSX Cluster Member is limited to 4096 interfaces.

By default, Virtual System can be connected to a maximum of 64 different Virtual Switches / Virtual Routers.
To be able up to configure more than 64 interfaces, you should use a bigger network mask for internal communication network. Therefore more VSs are possible here.


I see more problems in this theoretical discussion with CPU cores (CoreXL) and multi queueing and and and.

Please show me the appliance or open server with 4096 network interfaces or more than 64 cores. Okay, a 64K has a few more cores. But the discussion here is very theoretical😀

@Lari_Luoma 
With 250 VSs the tuning becomes interesting.😀

View solution in original post

0 Kudos
Reply
Lari_Luoma
Employee
Employee

You guys are correct. 250 Virtual Systems is a theoretical maximum for scalable platforms (44K/64K). However, it's not the recommendation and as Heiko mentioned, optimizing it could be a nightmare.. I have seen some environments with close to 100 VSs, but the real max. number I can recommend is about half of that. 50 VSs on one cluster is big environment that still works fine.

Of course it also depends on the load on the VSs. You can easily have 50 small VSs, but if each one of them is pushing one million connections you will reach the limit much earlier. 🙂 Here we come to one of my favorite discussions with customers... VSX was designed to have several small(ish) virtual firewalls. If you really need to have very big environments, consider using a physical devices for them. I have seen some customers running only one big VS per cluster. What's the point when you could use it as a regular gateway? 

If you have to ask "what's the maximum number of this or that" you most likely already have an architectural problem. 🙂

0 Kudos
Reply
Magnus-Holmberg
Advisor

Honestly i would like to say that 25-30 is more the limit.
we have a few clusters with 50 and they are very slow to make routing changes on in the VS.
Performance itself is no issue, but adding/changing interfaces routes takes several minutes instead of seconds.
During a maintenance window for a small change like changing default gw for a VS this is painfull.

We are currently running on R80.30 3.10.

Regards,
Magnus

https://www.youtube.com/c/MagnusHolmberg-NetSec
0 Kudos
Reply
FedericoMeiners
Advisor

I'll gladly share with you the lessons learned after I finish, I found that there are many "theorical" areas regarding VSX architecture that I'm looking to research 🙂

@HeikoAnkenbrand So far MultiQ is not an issue since it's configured in physical interfaces, most of our interfaces are VLANs therefore the MultiQ is only configured on 2 SFP links. ClusterXL it's a little bit trickier in large scale deployments of VSX but so far we are managing it.

Again, thank you all for your advises and thoughts, you are the best

Federico Meiners

____________
https://www.linkedin.com/in/federicomeiners/
0 Kudos
Reply