topic Re: 12600 with VSX low on memory in Firewall and Security Management

12600 with VSX low on memory

Maarten_Sjouw — Wed, 29 May 2019 22:33:50 GMT

Ran into a problem with the upgrade of a 12600 for a customer that was asked to assist. The setup of this customer was pretty simple, 2 management servers on R80.20 in HA. 2 x 12600 with R80.10 running VSX.

One piece of giveaway, one of the 12600's has 12GB memory the other (the backup) has 6GB memory

Now the challenge was to upgrade to R80.20 to be able to use the dynamic objects for Office 365.

So we start with the backup unit, there are 5 VS's and 55 virtual switches.

When done with the upgrade which went well (cpuse upgrade) we reboot the box and let it do it's things to see where we are I check with vsx stat -v and get a list of 39 problems like this:

Unable to open '/vs2/dev/fw0': Connection refused

Unable to open '/vs4/dev/fw0': Connection refused

Unable to open '/vs6/dev/fw0': Connection refused

Unable to open '/vs7/dev/fw0': Connection refused

Unable to open '/vs9/dev/fw0': Connection refused

Unable to open '/vs12/dev/fw0': Connection refused

Unable to open '/vs14/dev/fw0': Connection refused

On the console there were messages about SIC problems, we ended up doing a reinstall of the box with a USB stick and a clean R80.20, then ran a vsx_util reconfigure (after the base interface config) however the number of errors remains the same. Opened a TAC case, but nobody could find the cause of the messages and errors.

We decided to add more memory, so we sent 3 x 4GB onsite, but as the box has 2 physical CPU's it needs a even number of memory banks, so we put in 2 x 4GB to see if it would improve, it sure did, The number of problems went back to 20 with the added 2GB.

One other thing that was bothering me was the 55 Virtual Switches. The engineer that helped this customer during the first setup told the customer to create a vSwitch for each VLAN they use... 🤔

All these switches ended up in 1 trunk port and terminated a VLAN, out of the 55 there were 19 vSwitches that had no connection to any VS, so I tried to delete 1 that was all ok in SmartConsole, this went ok and got removed from both boxes. I continued to remove all the ones that had no issues. After a reboot the box came back without any of the previous errors. Then I could remove the last couple of unused vSwitches.

Then the local contact came back with 6 x 4GB DIMM's and put them all in, now the box is happily running with 24GB, why CP says it only supports 12 GB, I don't know.

We will see tomorrow that we upgrade the other box from R80.10 to R80.20 and also put more memory in them.

Re: 12600 with VSX low on memory

Maarten_Sjouw — Thu, 30 May 2019 14:45:40 GMT

Second box was upgraded without any problems at all. Customer decided not to upgrade the memory on that unit.

Re: 12600 with VSX low on memory

Lari_Luoma — Fri, 31 May 2019 01:31:56 GMT

Hi!

55 Virtual Switches definitely seems like a very ineffective architecture. There are two reasons to configure a Virtual Switch.

- When you want to share an interface/VLAN between several Virtual Systems

- When you want Virtual Systems to communicate directly (in this case no physical interface required for the VSW)

So, whoever told to create a virtual switch for every VLAN was wrong. 🙂 Especially if you have only 5 VSs, there is no reason that all VLANs would be shared between all VSs. If there was such a requirement, why to have a VSX in the first place. You could have simply used a security gateway with x number of VLAN-interfaces.

For the error you are seeing I'm not sure, but it could be related to the lack of memory. I've seen this error message before and reboot fixed it, so don't have the root cause.

Good luck!

Re: 12600 with VSX low on memory

Maarten_Sjouw — Fri, 31 May 2019 13:42:54 GMT

There were console messages about SIC and other errors in other places/moments.
I have added all the info into the case. I found in the end there were 36 vlan's used and out of them only 4 were used on multiple vs's.
So yeah there is a lot of work still to be done.

Re: 12600 with VSX low on memory

Lari_Luoma — Fri, 31 May 2019 17:20:05 GMT

Indeed, a lot of work. So those four VLANs would potentially require a VSW, but the rest no. 🙂

Re: 12600 with VSX low on memory

Kaspars_Zibarts — Wed, 12 Jun 2019 09:01:41 GMT

I saw the same approach with someone else here on forums when we took discussion offline.. Virtual switch for each VLAN! And it came from partner or CP themselves.. Someone is spreading a lot of b*#¤%&t! Sorry for swearing 🙂 I thought it would have been one off case but obviously not..

Re: 12600 with VSX low on memory

G_W_Albrecht — Wed, 12 Jun 2019 09:13:30 GMT

I also saw that live with a customer who was doing his own VSX design 😉 - only that vSwitches cost too much RAM he only found out later the hard way as some VSs could not work !

Re: 12600 with VSX low on memory

Maarten_Sjouw — Wed, 12 Jun 2019 09:34:30 GMT

Last week on the CPX NL I ran into a guy (from the same company that installed this system) who has one that has 200 V-Switches, he told me in the period this was installed there was a freelancer hired by the company, he had the advice to creata a switch for each VLAN, as you might run into the point that you need that VLAN on more than 1 VS.... 😞

Re: 12600 with VSX low on memory

Kenie634 — Thu, 13 Jun 2019 05:06:27 GMT

Customer decided not to upgrade the memory on that unit. Second box was upgraded without any problems at all.

Re: 12600 with VSX low on memory

Maarten_Sjouw — Mon, 24 Jun 2019 06:59:48 GMT

To continue this story, last week I tried to install the ongoing take 80, due to some issues we saw that were resolved with take 80. On the system that now has 24GB no problems at all, but the system with 12GB (the max according the specs) out of the 40, 12 Virtual Systems/Switches did not come in the ready state. Reverted the system to Take 47 and turned of Priority Queuing, as this was one of the reasons to install Take 80.
Looked at memory usage for VSX in cpview, this shows a per VS (system and switch) usage of 750MB per vs with Take 47 and 825MB per VS in Take 80. Due to the total of 40 VS's this makes the 12GB box run out of memory while starting each VS.

Re: 12600 with VSX low on memory

Maarten_Sjouw — Mon, 24 Jun 2019 12:33:29 GMT

Today another colleague told me this was propagated by Check Point in those days.