topic Re: R80.40 - Dynamic split of CoreXL in General Topics

R80.40 - Dynamic split of CoreXL

HeikoAnkenbrand — Mon, 22 Jun 2020 16:55:47 GMT

What is new in R80.40 EA.

A new interesting function for performance tuning has been included in R80.40. Dynamic split of CoreXL changes the assignment of CoreXL SND's and CoreXL firewall workers automatically without reboot.

How does this magic happens?

Adding and removing a CoreXL firewall worker
Adding and removing a CoreXL SND
Balance between CoreXL SND and CoreXL firewall worker
Work in ClusterXL environments
A reboot is not necessary

Pre-requisites:

GAIA 3.10 kernel (USFW/Kernel)
only Check Point appliances with 8 cores or more
currently supported on ClusterXL HA
currently VSLS is a limitation
In the following series all models are supported: 7000, 15000, 16000, 23000, 26000, 28000
In the following series only the listed models are supported:
- 5000: 5800 and 5900
- 6000: 6500, 6700, 6800 and 6900
Supported versions: Check Point R80.40 with Jumbo Hotfix Take 25 and above
CoreXL Dynamic Split does not support:
- Check Point Appliances that run in VSX mode (regardless of the number of CPU cores).
- Open Servers or Virtual Machines.
- Security Gateway (or Cluster Members) with Bridge interfaces.

How does it work?

Suppose we have two SND's and 6 CoreXL firewall workers. If no CoreXL SND's and CoreXL firewall workers are overloaded, nothing happens (picture 1).

Now, let's assume the CoreXL SNDs are overloaded (picture 2), a mathematical formula is used to calculate that a further CoreXL SND is added. In this case a CoreXL firewall worker 5 will not get any new connections (picture 3) and the connections are distributed to another CoreXL firewall worker for example to the CoreXL firewall worker 4. If there are no more connections running through this CoreXL firewall worker on core two, the core will be used for a new CoreXL SND instance (picture 4) . Now our appliance has three SND's and 5 CoreXL firewall workers.

It also works the other way round.

Picture 1 - nothing overloaded

Picture 2 - SND's overloaded

Picture 3 - CoreXL firewall worker stops the processing and distributes the connections.

Picture 4 - new SND is added

The Dynamic Split Daemon (dsd) has three stages in each iteration

1) Examine the current CPU utilization.
2) Decide if and what changes to make based on the current CPU utilization.
3) If needed, change the current CoreXL configuration

CLI Commands

In ClusterXL, you must configure all the Cluster Members in the same way. The dynamic_split command controls the Dynamic Split of CoreXL Firewall and SND instances on the local Security Gateway, or ClusterXL Member.

For more information, see R80.40 Performance Tuning Administration Guide - Chapter CoreXL.

Run these commands in the Expert mode

# dynamic_split

                            -o disable                 -> Disables the CoreXL Dynamic Split. Requires a reboot.
                            -o enable                  -> Enables the CoreXL Dynamic Split. Requires a reboot
                            -o start                      -> Starts the CoreXL Dynamic Split after it was stopped.
                            -o stop                       -> Stops the CoreXL Dynamic Split. This change survive the reboot.
-p -> Show status

Re: R80.40 EA - Dynamic split of CoreXL

HeikoAnkenbrand — Mon, 11 Nov 2019 16:25:48 GMT

A small update of the article with pictures.

Re: R80.40 EA - Dynamic split of CoreXL

rolf — Wed, 13 Nov 2019 08:56:01 GMT

Nice info!

Re: R80.40 EA - Dynamic split of CoreXL

HeikoAnkenbrand — Tue, 28 Jan 2020 18:47:51 GMT

Update: CLI CommandsCLI CommandsCLI Commands

Re: R80.40 - Dynamic split of CoreXL

phlrnnr — Thu, 30 Jan 2020 16:08:42 GMT

Is this enabled by default in R80.40? Or does it have to be turned on?

Re: R80.40 - Dynamic split of CoreXL

Ilya_Yusupov — Thu, 30 Jan 2020 17:35:09 GMT

@phlrnnr - It have to be turned on.

Re: R80.40 - Dynamic split of CoreXL

HeikoAnkenbrand — Thu, 30 Jan 2020 21:05:31 GMT

I did a cluster update to R80.40 today and have it enabled on with 16 core.

Unfortunately I cannot test it, because the cores only had a utilisation of about 10%:-)

For more information, see R80.40 Performance Tuning Administration Guide - Chapter CoreXL.

Run these commands in the expert mode

# dynamic_split

                            -o disable                 -> Disables the CoreXL Dynamic Split. Requires a reboot.
                            -o enable                  -> Enables the CoreXL Dynamic Split. Requires a reboot
                            -o start                      -> Starts the CoreXL Dynamic Split after it was stopped. This change survives the reboot-
                            -o stop                       -> Stops the CoreXL Dynamic Split. This change does not survive the reboot.

-p -> Show status

Re: R80.40 - Dynamic split of CoreXL

HeikoAnkenbrand — Fri, 31 Jan 2020 07:25:44 GMT

I added that to the original article.

Re: R80.40 - Dynamic split of CoreXL

Florian_Winterb — Tue, 04 Feb 2020 17:35:31 GMT

What are the correct steps?

first -> enable

second -> start

Re: R80.40 - Dynamic split of CoreXL

Harry_Morgan — Wed, 05 Feb 2020 17:16:21 GMT

If this function is activated for r80.40 with 8 cores by default?

Re: R80.40 - Dynamic split of CoreXL

HeikoAnkenbrand — Wed, 18 Mar 2020 19:55:35 GMT

Yes, it is enabled with 8 and more cores by default.

Re: R80.40 - Dynamic split of CoreXL

HristoGrigorov — Thu, 19 Mar 2020 04:34:31 GMT

Any study on how efficient actually is this CoreXL split ? Also, do you know how often is current load evaluated and re-assignment made ?

Re: R80.40 - Dynamic split of CoreXL

Pablo_Montega — Sat, 18 Apr 2020 06:05:56 GMT

I'm using it on a 16 core system. I don't see any redistribution of cores.

Re: R80.40 - Dynamic split of CoreXL

Dorit_Dor — Sat, 18 Apr 2020 06:19:52 GMT

Did you turned it on (As mentioned above It have to be turned on).

The function is off by default (initially). It enables us to get wide production exposure before exposing everyone to the new functionality

Re: R80.40 - Dynamic split of CoreXL

Jerry — Sat, 18 Apr 2020 07:36:32 GMT

here is a bit I'm about to turn on about, just waiting for "reboot time" 🙂

[Expert@cp:0]# uname -a
Linux cp 3.10.0-957.21.3cpx86_64 #1 SMP Mon Jan 6 17:24:28 IST 2020 x86_64 x86_64 x86_64 GNU/Linux
[Expert@cp:0]# dynamic_split -P
P is not a valid option
Usage: enable or disable, stop or start [-o enable|disable|stop|start]
print status [-p]

[Expert@cp:0]# dynamic_split -p
Dynamic Split is currently off
ALPHA: 10
EMERGENCY_CPU_HANDLING_THRESHOLD: 40

Re: R80.40 - Dynamic split of CoreXL

Jerry — Sat, 18 Apr 2020 07:40:15 GMT

I'm using it on 24-core 15600 CXL and it makes massive difference only when SXL has more than 90% of rates so most of your SecureXL traffic hits proper templates 😛 will post something soon how this performs but I see a nice way of self-disti within CXL/SND on that ClusterXL.

Re: R80.40 - Dynamic split of CoreXL

Jerry — Sat, 18 Apr 2020 08:16:44 GMT

sequence of events though:

[Expert@cp:0]# cat /opt/CPsuite-R80.40/fw1/log/dynamic_split.elg
[Sat Apr 18 08:34:12 BST 2020] Dynamic Split is currently off ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:36:58 BST 2020] Dynamic Split is currently off ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:51:18 BST 2020] spreading queues
[Sat Apr 18 08:51:18 BST 2020] sorted cpus aquired
[Sat Apr 18 08:51:24 BST 2020] ON
[Sat Apr 18 08:56:16 BST 2020] Dynamic Split is currently on ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:57:37 BST 2020] Dynamic Split is currently on ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:57:58 BST 2020] OFF due to disablement
[Sat Apr 18 08:57:58 BST 2020] weights reset
[Sat Apr 18 08:57:58 BST 2020] insts started
[Sat Apr 18 08:57:58 BST 2020] insts affined
[Sat Apr 18 08:58:02 BST 2020] snds reset
[Sat Apr 18 08:58:02 BST 2020] state file removed
[Sat Apr 18 09:06:30 BST 2020] starting
[Sat Apr 18 09:06:30 BST 2020] ON following "-o start"
[Sat Apr 18 09:11:59 BST 2020] spreading queues
[Sat Apr 18 09:11:59 BST 2020] sorted cpus aquired
[Sat Apr 18 09:12:04 BST 2020] ON

+ following CCC:

[Executing:]# fw ctl affinity -l -a
Kernel fw_0: CPU 23
Kernel fw_1: CPU 11
Kernel fw_2: CPU 22
Kernel fw_3: CPU 10
Kernel fw_4: CPU 21
Kernel fw_5: CPU 9
Kernel fw_6: CPU 20
Kernel fw_7: CPU 8
Kernel fw_8: CPU 19
Kernel fw_9: CPU 7
Kernel fw_10: CPU 18
Kernel fw_11: CPU 6
Kernel fw_12: CPU 17
Kernel fw_13: CPU 5
Kernel fw_14: CPU 16
Kernel fw_15: CPU 4
Kernel fw_16: CPU 15
Kernel fw_17: CPU 3
Kernel fw_18: CPU 14
Kernel fw_19: CPU 2
Interface eth1-01: has multi queue enabled *** 10G SFP+
Interface eth1-02: has multi queue enabled *** 10G SFP+

🙂 any thoughts though?

Cheers mates!

Jerry

Re: R80.40 - Dynamic split of CoreXL

Silvia_Day — Fri, 08 May 2020 07:33:23 GMT

Nice info!

Re: R80.40 - Dynamic split of CoreXL

_Val_ — Fri, 08 May 2020 09:23:52 GMT

Hi @HeikoAnkenbrand, a smal correction:

You say, -o stop command does not survive reboot.

The Admin guide says otherwise.

Re: R80.40 - Dynamic split of CoreXL

HeikoAnkenbrand — Fri, 08 May 2020 11:06:22 GMT

Was different in R80.40 EA and the presentations in Israel.

Thanks, I'll change that.