dynamic balancing, VSX and core affinity

[Wolfgang]

I need some advice and real world experience from the field.

In VSX environments you have to find a good distribution of your cores to virtual systems. Experience over the years shows it's a good configuration to allocate dedicated cores to heavy used VS. You have to have a look at all your cores and you have to play a little bit with the distribution between cores, VS and processes. But it works.

Now we have Dynamic Balancing for CoreXL with support for VSX. Anything changes with this ?

Can I set all my cores to shared for all VS and dynamic balancing did the work and will distribute everything balanced ?

Maybee it's better to disable dynamic balancing and set affinity with dedicated cores for VS ?

Great Security Gateway Performance Optimization - VSX  in the past initiated by @Kaspars_Zibarts but it would be interesting with an update regarding dynamic balancing.

[AmitShmuel]

Few thing to note:

1. These symptoms may only occur on large systems with many VSs
2. As said, applying the private hotfix solves it
3. The fix will be published as part of R81.10 JHF #7, which is due to next week
   1. For R81 - it will be part of JHF #11, which is also due to next week
   2. For R80.40 - it is part of JHF #19, which was already released

[genisis__]

Thanks Amit.

Also may be an idea to reference the JHFA take revisions as the above is confusing:

R80.40 latest revision is Take 180

R81 latest revision is Take 74

R81.10 latest revision is Take 78

[AmitShmuel]

Thanks.

R80.40 JHF #19 is indeed Take 180

For R81 and R81.10, Takes are not yet determined.

[Alex-]

I'm running a large VSX environment (25+ VS w/ VSLS) on high-end CP appliances and it doesn't work. I enable dynamic split following the SK, wait for confirmation, reboot then cpview shows Dynamic Balancing Status Initializing then On. After a few minutes, it's back in off status and with 2 SND out of 72 CPU in a loaded environment. Same situation on the secondary cluster member. I'm currently assigning SND with fw ctl affinity after having disabled dynamic_split. R80.40 Take 173, planning to go to R81.10 when the next GA HF is released.

[genisis__]

 Alex - this is the same symptom I experienced.

did you try dynamic_balancing -r  (This worked for bit for me, then reverted back to a couple of cores being stuck in the other state (check cpview)

additionally review the following logs (TAC will need these)

$FWDIR/log/dsd.elg

$FWDIR/log/dynamic_split.elg

[Alex-]

Thanks for the tips genisis, appreciated. No I didn't try the flag as it wasn't clear from the SK it was needed and enabling didn't provide any warnings. Also, I have limited maintenance windows for that cluster given its size and configuration (dynamic routing & co) and had enough reboots as it is. 😄

Manual affinity does the job for now, I will wait R81.10 to explore this feature again.

[genisis__]

I believe dynamic balancing on R81.10 with JHFA75 or above should be good.
# dynamic_balancing -h
-h - Shows built-in help
-o enable - Enables the CoreXL Dynamic Balancing (requires a reboot)
-o disable - Disables the CoreXL Dynamic Balancing (requires a reboot)
-o stop - Stops the CoreXL Dynamic Balancing ("freezes" it; survives a reboot)
-o start - Starts the CoreXL Dynamic Balancing ("resumes" it; survives a reboot)
-p - Shows the current state of the CoreXL Dynamic Balancing
-r - Resets the CoreXL configuration to the default and keeps the CoreXL Dynamic Balancing enabled

[AmitShmuel]

Hi, Take 180 has it fixed.