Security Gateway Performance Optimization - VSX

[Kaspars_Zibarts]

This is a feedback to the awesome session by Tim Hall TechTalk: Security Gateway Performance Optimization with Tim Hall - was great presentation. Thanks heaps 

I thought that it would be worthwhile adding some comments regarding VSX as it has it's own little idiosyncrasies when it comes to SXL, CoreXL, SMT and MQ. In general every technology point that was discussed by Tim actually applies to VSX too and it is vital to understand all of them before you start digging into VSX tuning.

As always - VSX is deployed in so many different ways that making general recommendations is impossible: this is merely some "gotcha's" learned over years using it. 

AGGRESSIVE AGING

Watch out for under-dimensioned VS. By default connections table is set to 15000 concurrent connections that is rather low. When AA turns on, it may impact working traffic, i.e. we had issues with some very specific RDP running over SSL when aggressive aging kicked in. Keep close eye on fw ctl pstat

64 BIT VS SUPPORT

Make sure that you are utilising R80+ in full by switching (if not done by default) VSes to 64 bit support thus avoiding memory shortage for connections

COREXL - FW WORKER RESOURCE POOLS

By default, all your VSes will share the same resource pool of fw workers. On one hand it is "an easy maintenance" approach, but if you have critical VSes, I would recommend to protect them by allocating them dedicated CPU resources to avoid situations with "elephant connections" and/or "elephant VSes" killing your critical resource access.

Default split between SXL (0-3) and CoreXL (4-23) on VSX

And dedicated approach: VS0,1,2,5 on core 4, VS3,4,7 on core 7, VS6 on 12-21 and VS8 - using default all. It's just an example!

COREXL - "MAPPING" CORES

If you do decide to go with strict resource allocation, i.e dedicated CPU cores for each VS, create a "map" of your CPUs to visualize allocation. Especially important with hyper-threaded CPU cores as numbering is not sequential anymore. Couple of rules to keep in mind to keep best performance

• do not split one VS across two physical cores (look at VS4 and VS5 below)
• allocate both "main" core and it's HT sibling, not just main cores or HT instances (look below for example VS0 has two cores 4 and 28 allocated, not 4 and 5)
• highly loaded VSes keep on the same physical CPU as SXL as you may gain from caching
• we noticed best performance when we had one to one mapping of a specific fw workers on VS to a CPU core - but that can eat up all your available cores quickly.

Below is a sample of 48 core hyper-threaded system map

COREXL - IDENTITY AWARENESS LARGE DEPLOYMENTS

We learned hard away, especially prior R80.10 release that pepd and pdpd can go "nuts" and consume a lot of CPU resources, therefore to avoid any impact on fw workers we put them on dedicated cores

I realise that I'm digging my own grave here  but I'll probably learn something new soo

[Ian_Geraris]

Hi Kaspars, a very interesting post indeed!!

I was a bit of a late reader, but I do think I have a couple of questions... in case you or anyone else can tackle...

1) If you indeed go with the dedicated approach that you propose, with various VSystems getting allocated to different Core pools... Say in a VSLS environment, where a particular VSystem can be active on any of the cluster-members, wouldn't that in essence limit the amount of cores that the particular VSystem will be getting and also render some of the cores needlessly idle?

2) I'm pretty sure I've read somewhere that whenever Hyper Threading is enabled, the HT cores will always be getting the second half of Core numbers assigned to them, or something... Was that indeed the rule that you also followed in order to create this core map of yours?

Cheers!

[Kaspars_Zibarts]

Hi Ian

you are absolutely correct regarding point 1 - it's a catch 22 situation so you have to make your own call depending on resources you are trying to protect. If you share all cores to all VSes you may face a situation where one "elephant" flow on one VS will fully utilise one (or more) core for example. That will impact connections on all other VSes that will try to use the same core as VSX does not have dynamic dispatcher so chances are quite high. In nutshell - I try to continuously monitor core utilization and adjust if necessary to have reasonable utilization across all yet having some protection from one VS impacting rest.

Correct about #2 - if HT is enabled, by default cores will be allocated to SXL and CoreXL in corresponding sibling "pairs". So you can always use that as "basis" when creating own maps

[Ian_Geraris]

Thanks for the clarifications Kaspars... your approach is also quite interesting!!

I think i'm going to stick to the "shared cores" approach among all VSystems for my VSLS environments - at least for the time being - since the devices that we have are quite big for the amount of traffic that is supposed to be passing through them and the fact that I just don't want to be worrying about having different cores sitting idle on different boxes - based on which box the various VSystems are going to be active on, if I start statically allocating them to various cores...

The only affinities that i'm actually thinking of "playing" with in a VSLS environment, would be those of the Mgmt & Sync interfaces, as well as that of VS0. In an effort to maybe isolate those to one or two specific cores and "protect" them from the other SXL or CXL instances... not sure though if this would also lead to rendering the particular cores pretty much idle on some of the boxes (we mainly have clusters of 4 boxes).

All in all, I think that VSLS and the various core affinities approaches is a "nerve-racking" topic  

Cheers!

[Kaspars_Zibarts]

Totally agree - that's how we started, by isolating VS0, Mgmt and Sync! And then it went on and grew little bigger  has proven itself effective on some occasions, I must admit

[Timothy_Hall]

Great info Kaspars Zibarts, you may want to consider talking about this topic at CPX:

https://community.checkpoint.com/community/about-checkmates/blog/2018/09/12/call-for-papers-cpx-360-... 

Deadline for submissions is Nov 15th.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Kaspars_Zibarts]

 let me read T&Cs - I'm not the best "stage/public" speaker. But maybe it's worth trying!

[Jelle_Hazenberg]

Hi Kaspars,

Thanks for sharing such useful information. I always find it hard to find such in depth articles about "how VSX works" and "what you should check" (do you have more articles that i can read?) but that's not what i want to ask you, actually i have a question about the 64 bit VSes.

 I just checked our environment and discovered that our VSes are running 32 bit...

We are running R80.10 and we own 2 x 12600 appliance which has the configuration option "set edition 64-bit".

Could you please explain why our VSes are running on 32 bit and how to let them run on 64 bit?

Kind regards,

Jelle

[Kaspars_Zibarts]

Yep - the trick with R80.10 is that even your underlying GAIA is set to 64 bit edition as in your example, actual VS kernel by default is set to 32 bits, so you must change it manually to 64bit. I really don't know why and have no answer to that, I even had a little whinge here https://community.checkpoint.com/thread/9762-vsxutil-reconfigure-wishlist  but as it goes apparently it's a default setting in R80.20