dynamic balancing, VSX and core affinity

[Wolfgang]

I need some advice and real world experience from the field.

In VSX environments you have to find a good distribution of your cores to virtual systems. Experience over the years shows it's a good configuration to allocate dedicated cores to heavy used VS. You have to have a look at all your cores and you have to play a little bit with the distribution between cores, VS and processes. But it works.

Now we have Dynamic Balancing for CoreXL with support for VSX. Anything changes with this ?

Can I set all my cores to shared for all VS and dynamic balancing did the work and will distribute everything balanced ?

Maybee it's better to disable dynamic balancing and set affinity with dedicated cores for VS ?

Great Security Gateway Performance Optimization - VSX  in the past initiated by @Kaspars_Zibarts but it would be interesting with an update regarding dynamic balancing.

[Chris_Atkinson]

Setting static affinity is less relevant in current versions in my opinion but there are still some exceptions such as scenarios that may benefit from HyperFlow in future.

[Kaspars_Zibarts]

Yeah. I don't if I would dare to allow dynamic split on our core internal VSX. Too much at stake there. And VSes are really different in sizes. I'm afraid that it may lead to situation where a smaller VS can trigger some unwanted split changes leading to outages on bigger VSes.

But that's just me being old fashioned 👴

Probably would risk it in "provider" like environment with more equally sized VSes.

In all honesty we have not been in such situation where dynamic split would have saved us.

To give another scenario where one of external VSes got resources exhausted because of DDOS attack - I'm curious how would dynamic split would work in this scenario - just keep allocating FWKs till SNDs too are overloaded? 🙂

Too many questions to be honest. I'll let someone else to experiment with it in production 🙂

[Wolfgang]

I‘m with you @Kaspars_Zibarts most of the time the old fashioned way will be the save way and let you sleep at night. But sometimes new features are really useful and save a lot of time.

Me too, I don‘t want to be the first to try. The systems we could try this have to be run as stable as possible. Possibly someone here tried and will share his experience 😀

[genisis__]

I agree - too many unknowns, and for control freaks like us we want predictable, measurable statics per VS so capacity can be managed correctly.

In a traditional gateway scenario, I would problem enable it, on a fresh installation.

[_Val_]

@Chen_Muchtar  can you please advise?

[AmitShmuel]

Similarly to SGW, Dynamic Balancing aims to balance the load between the FWKs cores and the SNDs cores.

A prerequisite to start Dynamic Balancing, is having all FWKs set to the default FWKs CPUs (for example in an 8 cores machine, 2-7).

Upon detecting an imbalance (SNDs working harder), Dynamic Balancing will set all VSs FWKs to a smaller set of CPUs, and have SNDs take over the CPU.

Average load calculation remain the same, Dynamic Balancing discards any outliers CPUs that may be working harder due to some specific VS.

I'd be happy to review your advanced configuration and share my feedback, feel free to contact me at amitshm@checkpoint.com.

Thanks,
Amit

[Cristian_F_CCSM]

Hello, the situation vs 1 linked to internet and vs 2 as ISFW with CoreXL Dynamic balancing enabled is very intesting. In DDOS case from internet, i would't that there is an impact on vs 2 ISFW. How about this situation?

Second question, if I enable CoreXL Dynamic Balancing, the "fw workers" number is still editable from VS configuration on SmartConsole. I would have expected that this type of configuration is disabled after Dynamic Balancing activation. Can you explain (or copy a link) how interact these two feature / configuration please?

Thanks

[AmitShmuel]

The number of FW workers is still editable.

On VSX, Dynamic Balancing only changes the amount of cores running FW workers, so you can configure any number of them.
Upon SND addition, it will set the FWKs of all VSs to the new set of cores.

Here is an example:

Default state:
Core 0: SND
Core 1-3: FWKs (fwk0_0, fwk0_1, ..., fwk1_0, fwk1_1, ..., fwkN_N)

Dynamic Balancing adds SND:
Core 0: SND
Core 1: SND
Core 2-3: FWKs (fwk0_0, fwk0_1, ..., fwk1_0, fwk1_1, ..., fwkN_N)

[CheckPointerXL]

Hello Amit,

This is a great explanation.

What about reverse situation? A VS needs more fw worker instances: could they get othere cores for this purpose? I'm confused about this because after enabling Dynamic Balancing i can still edit from SmartConsole number of CoreXL.

I guess, if VS needs more fw worker, will they be assigned to VS anyway? So, i will have 10 fw worker on a VS with 8 CoreXL sticked on smartconsole, is this possibile?

I m confused

[AmitShmuel]

By reverse situation, do you mean more FW instances, or more FW cores?

Dynamic Balancing can reduce the number of SND cores to utilize more FW cores, but since changing the number of FW workers results in VS restart, it is not possible to do it dynamically, hence it can only allocate more cores to the same amount of FW workers, leaving the minimum number for SNDs of course.

[CheckPointerXL]

Ok, so let's assume I have a VSX GW with 32 core.

Then, I create a VS and I configure 10 CoreXL instances on CoreXL tab inside the VS Object in SmartConsole.

The dynamic balancing happens INSIDE that 10 cores, by balancing SND/Worker on that 10 core, is this right?

By reading documentation i Understood that with dynamic balancing one single VS could potentially use all 32 cores.

[AmitShmuel]

There is a complete separation between FW cores and SND cores (see my illustration above).

The dynamic balancing happens on the system as a whole, if more FW cores are needed, it will allocate more FW cores, and vice versa.

The configured 10 CoreXL instances will run freely on the FW cores (whether there are 2 cores, 20 cores, or any other number).
The number of FW cores is usually determined by the number of SND cores. i.e. you have 4 SNDs on a 32 cores machine, then you'll have 28 cores used for FW.

[CheckPointerXL]

Amil, thank you very much for your explanation.

Now i understand that configuring 10 CoreXL Fw worker on SmartConsole it means that these worker can "join" 20 logical core or also 5; it helped me the attached illustration.

[Cristian_F_CCSM]

Hello, OK, clear about the second question.

About the first scenario indicated (two vs: external and internal fw), do you have experience?

Thanks

[AmitShmuel]

Can you please elaborate on that scenario? what is the concern here?

[Cristian_F_CCSM]

Hello, yes sure, my doubt is with this situation:

- vs 1 (internet) receive a DDOS attack

- The requests to CPUs (for IPS, logging etc.) are high

- Dynamic balancing is enabled

In this case the vs1 and vs2 use the same CPUs fore CoreXL and, therefore, in this case, will be some issue also for vs2 (internal) and not only vs1 (internet)?!

If we assign some CPUs for vs 1 and, others CPUs to vs2, we can reduce this type of risk.

In this scenario, with dynamic balancig enabled, VSLS can minimize the risk (in my humble opinion).

Regards

[AmitShmuel]

Current Dynamic Balancing implementation uses all FWK CPUs for all VSs, similar to the default out of the box configuration.

Are you suggesting to separately assign VSs CPUs in advance, or on the fly?

[Cristian_F_CCSM]

Hello, to prevent the described problem i prefer configure static CPU affinity during the VSX GW first configuration.

[Piet_vd_Maas]

My experience with Dynamic Balancing is quite positive.

The load on all CPU's is better balanced as it should. And when the load of a interface is heavier the impact on the throughput is lower. 

[genisis__]

I'm running dynamic balancing on a R81.10 VSX system with JHFA66, however additional private hotfix was required, which also contained fixes for dynamic balancing.

Symptoms I experinced:

- Dynamic balancing initially starts, but once all the VS have been fully loaded it turns off.

- Cores get stuck in OTHER state as they are transitioning between fwk and SND.

Please note that I've discussed these with TAC and if not done so already the fixes related to dynamic balancing will be integrated (I think they were introduced in JHFA75).

[Kaspars_Zibarts]

Thanks for keeping us in the loop! Really appreciated. 

[genisis__]

Additional info which I think is useful, which as sent by the TAC engineer:

PRHF-25607  - is going to be part of next Jumbo PRJ-41482 
PRHF-25610  - ready for commit state, trying to push this one as well as soon as possible PRJ-41634 
PRHF-25603  - already part of the Jumbo take 75 - PRJ-39820 
PRHF-25597  - already part of Jumbo 75 PRJ-39324 
PRHF-25594  - ready for commit state, trying to push this one as well as soon as possible PRJ-41124 
PRHF-25611  – going to be part of next jumbo 

[_Val_]

@genisis__ could you please send me offline the contact of that TAC engineer? Thanks in advance, Val

[genisis__]

Will do.

[Naama_Specktor]

@genisis__ 

Hi 🙂

My name is Naama Specktor and I am checkpoint employee,

I will appreciate it if you will share TAC SR # with me , here or in PM.

thanks!

Naama 

[genisis__]

Ping Val - already sent info to him.

[AmitShmuel]

Few thing to note:

1. These symptoms may only occur on large systems with many VSs
2. As said, applying the private hotfix solves it
3. The fix will be published as part of R81.10 JHF #7, which is due to next week
   1. For R81 - it will be part of JHF #11, which is also due to next week
   2. For R80.40 - it is part of JHF #19, which was already released

[genisis__]

Thanks Amit.

Also may be an idea to reference the JHFA take revisions as the above is confusing:

R80.40 latest revision is Take 180

R81 latest revision is Take 74

R81.10 latest revision is Take 78

[AmitShmuel]

Thanks.

R80.40 JHF #19 is indeed Take 180

For R81 and R81.10, Takes are not yet determined.