topic Re: dynamic balancing, VSX and core affinity in Firewall and Security Management

dynamic balancing, VSX and core affinity

Wolfgang — Thu, 14 Apr 2022 13:22:14 GMT

I need some advice and real world experience from the field.

In VSX environments you have to find a good distribution of your cores to virtual systems. Experience over the years shows it's a good configuration to allocate dedicated cores to heavy used VS. You have to have a look at all your cores and you have to play a little bit with the distribution between cores, VS and processes. But it works.

Now we have Dynamic Balancing for CoreXL with support for VSX. Anything changes with this ?

Can I set all my cores to shared for all VS and dynamic balancing did the work and will distribute everything balanced ?

Maybee it's better to disable dynamic balancing and set affinity with dedicated cores for VS ?

Great Security Gateway Performance Optimization - VSX in the past initiated by @Kaspars_Zibarts but it would be interesting with an update regarding dynamic balancing.

Re: dynamic balancing, VSX and core affinity

Chris_Atkinson — Thu, 14 Apr 2022 13:29:23 GMT

Setting static affinity is less relevant in current versions in my opinion but there are still some exceptions such as scenarios that may benefit from HyperFlow in future.

Re: dynamic balancing, VSX and core affinity

Kaspars_Zibarts — Thu, 14 Apr 2022 13:36:15 GMT

Yeah. I don't if I would dare to allow dynamic split on our core internal VSX. Too much at stake there. And VSes are really different in sizes. I'm afraid that it may lead to situation where a smaller VS can trigger some unwanted split changes leading to outages on bigger VSes.

But that's just me being old fashioned 👴

Probably would risk it in "provider" like environment with more equally sized VSes.

In all honesty we have not been in such situation where dynamic split would have saved us.

To give another scenario where one of external VSes got resources exhausted because of DDOS attack - I'm curious how would dynamic split would work in this scenario - just keep allocating FWKs till SNDs too are overloaded? 🙂

Too many questions to be honest. I'll let someone else to experiment with it in production 🙂

Re: dynamic balancing, VSX and core affinity

_Val_ — Thu, 14 Apr 2022 15:13:02 GMT

@Chen_Muchtar can you please advise?

Re: dynamic balancing, VSX and core affinity

Wolfgang — Thu, 14 Apr 2022 19:44:09 GMT

I‘m with you @Kaspars_Zibarts most of the time the old fashioned way will be the save way and let you sleep at night. But sometimes new features are really useful and save a lot of time.

Me too, I don‘t want to be the first to try. The systems we could try this have to be run as stable as possible. Possibly someone here tried and will share his experience 😀

Re: dynamic balancing, VSX and core affinity

genisis__ — Thu, 14 Apr 2022 20:12:42 GMT

I agree - too many unknowns, and for control freaks like us we want predictable, measurable statics per VS so capacity can be managed correctly.

In a traditional gateway scenario, I would problem enable it, on a fresh installation.

Re: dynamic balancing, VSX and core affinity

AmitShmuel — Thu, 14 Apr 2022 22:07:35 GMT

Similarly to SGW, Dynamic Balancing aims to balance the load between the FWKs cores and the SNDs cores.

A prerequisite to start Dynamic Balancing, is having all FWKs set to the default FWKs CPUs (for example in an 8 cores machine, 2-7).

Upon detecting an imbalance (SNDs working harder), Dynamic Balancing will set all VSs FWKs to a smaller set of CPUs, and have SNDs take over the CPU.

Average load calculation remain the same, Dynamic Balancing discards any outliers CPUs that may be working harder due to some specific VS.

I'd be happy to review your advanced configuration and share my feedback, feel free to contact me at amitshm@checkpoint.com.

Thanks,
Amit

Re: dynamic balancing, VSX and core affinity

Cristian_F_CCSM — Mon, 19 Sep 2022 14:25:15 GMT

Hello, the situation vs 1 linked to internet and vs 2 as ISFW with CoreXL Dynamic balancing enabled is very intesting. In DDOS case from internet, i would't that there is an impact on vs 2 ISFW. How about this situation?

Second question, if I enable CoreXL Dynamic Balancing, the "fw workers" number is still editable from VS configuration on SmartConsole. I would have expected that this type of configuration is disabled after Dynamic Balancing activation. Can you explain (or copy a link) how interact these two feature / configuration please?

Thanks

Re: dynamic balancing, VSX and core affinity

AmitShmuel — Mon, 19 Sep 2022 14:50:46 GMT

The number of FW workers is still editable.

On VSX, Dynamic Balancing only changes the amount of cores running FW workers, so you can configure any number of them.
Upon SND addition, it will set the FWKs of all VSs to the new set of cores.

Here is an example:

Default state:
Core 0: SND
Core 1-3: FWKs (fwk0_0, fwk0_1, ..., fwk1_0, fwk1_1, ..., fwkN_N)

Dynamic Balancing adds SND:
Core 0: SND
Core 1: SND
Core 2-3: FWKs (fwk0_0, fwk0_1, ..., fwk1_0, fwk1_1, ..., fwkN_N)

Re: dynamic balancing, VSX and core affinity

Cristian_F_CCSM — Mon, 19 Sep 2022 14:59:22 GMT

Hello, OK, clear about the second question.

About the first scenario indicated (two vs: external and internal fw), do you have experience?

Thanks

Re: dynamic balancing, VSX and core affinity

Piet_vd_Maas — Mon, 19 Sep 2022 15:05:19 GMT

My experience with Dynamic Balancing is quite positive.

The load on all CPU's is better balanced as it should. And when the load of a interface is heavier the impact on the throughput is lower.

Re: dynamic balancing, VSX and core affinity

CheckPointerXL — Mon, 19 Sep 2022 16:02:19 GMT

Hello Amit,

This is a great explanation.

What about reverse situation? A VS needs more fw worker instances: could they get othere cores for this purpose? I'm confused about this because after enabling Dynamic Balancing i can still edit from SmartConsole number of CoreXL.

I guess, if VS needs more fw worker, will they be assigned to VS anyway? So, i will have 10 fw worker on a VS with 8 CoreXL sticked on smartconsole, is this possibile?

I m confused

Re: dynamic balancing, VSX and core affinity

AmitShmuel — Mon, 19 Sep 2022 22:26:12 GMT

By reverse situation, do you mean more FW instances, or more FW cores?

Dynamic Balancing can reduce the number of SND cores to utilize more FW cores, but since changing the number of FW workers results in VS restart, it is not possible to do it dynamically, hence it can only allocate more cores to the same amount of FW workers, leaving the minimum number for SNDs of course.

Re: dynamic balancing, VSX and core affinity

AmitShmuel — Mon, 19 Sep 2022 22:27:34 GMT

Can you please elaborate on that scenario? what is the concern here?

Re: dynamic balancing, VSX and core affinity

CheckPointerXL — Mon, 19 Sep 2022 22:48:29 GMT

Ok, so let's assume I have a VSX GW with 32 core.

Then, I create a VS and I configure 10 CoreXL instances on CoreXL tab inside the VS Object in SmartConsole.

The dynamic balancing happens INSIDE that 10 cores, by balancing SND/Worker on that 10 core, is this right?

By reading documentation i Understood that with dynamic balancing one single VS could potentially use all 32 cores.

Re: dynamic balancing, VSX and core affinity

AmitShmuel — Tue, 20 Sep 2022 09:02:15 GMT

There is a complete separation between FW cores and SND cores (see my illustration above).

The dynamic balancing happens on the system as a whole, if more FW cores are needed, it will allocate more FW cores, and vice versa.

The configured 10 CoreXL instances will run freely on the FW cores (whether there are 2 cores, 20 cores, or any other number).
The number of FW cores is usually determined by the number of SND cores. i.e. you have 4 SNDs on a 32 cores machine, then you'll have 28 cores used for FW.

Re: dynamic balancing, VSX and core affinity

CheckPointerXL — Wed, 21 Sep 2022 09:33:32 GMT

Amil, thank you very much for your explanation.

Now i understand that configuring 10 CoreXL Fw worker on SmartConsole it means that these worker can "join" 20 logical core or also 5; it helped me the attached illustration.

Re: dynamic balancing, VSX and core affinity

Cristian_F_CCSM — Wed, 21 Sep 2022 14:48:13 GMT

Hello, yes sure, my doubt is with this situation:

- vs 1 (internet) receive a DDOS attack

- The requests to CPUs (for IPS, logging etc.) are high

- Dynamic balancing is enabled

In this case the vs1 and vs2 use the same CPUs fore CoreXL and, therefore, in this case, will be some issue also for vs2 (internal) and not only vs1 (internet)?!

If we assign some CPUs for vs 1 and, others CPUs to vs2, we can reduce this type of risk.

In this scenario, with dynamic balancig enabled, VSLS can minimize the risk (in my humble opinion).

Regards

Re: dynamic balancing, VSX and core affinity

AmitShmuel — Wed, 21 Sep 2022 15:52:05 GMT

Current Dynamic Balancing implementation uses all FWK CPUs for all VSs, similar to the default out of the box configuration.

Are you suggesting to separately assign VSs CPUs in advance, or on the fly?

Re: dynamic balancing, VSX and core affinity

Cristian_F_CCSM — Wed, 21 Sep 2022 16:51:35 GMT

Hello, to prevent the described problem i prefer configure static CPU affinity during the VSX GW first configuration.