R80.40 - Dynamic split of CoreXL

[HeikoAnkenbrand]

What is new in R80.40 EA.

A new interesting function for performance tuning has been included in R80.40. Dynamic split of CoreXL changes the assignment of  CoreXL SND's and CoreXL firewall workers automatically without reboot.

How does this magic happens?

• Adding and removing a CoreXL firewall worker
• Adding and removing a CoreXL SND
• Balance between CoreXL SND and CoreXL firewall worker
• Work in ClusterXL environments
• A reboot is not necessary
  

Pre-requisites:

• GAIA 3.10 kernel (USFW/Kernel)
• only Check Point appliances with 8 cores or more
• currently supported on ClusterXL HA
• currently VSLS is a limitation
• In the following series all models are supported: 7000, 15000, 16000, 23000, 26000, 28000
• In the following series only the listed models are supported:
  
  • 5000: 5800 and 5900
  • 6000: 6500, 6700, 6800 and 6900
• Supported versions: Check Point R80.40 with Jumbo Hotfix Take 25 and above

• CoreXL Dynamic Split does not support:

  • Check Point Appliances that run in VSX mode (regardless of the number of CPU cores).

  • Open Servers or Virtual Machines.

  • Security Gateway (or Cluster Members) with Bridge interfaces.

How does it work?

Suppose we have two SND's and 6 CoreXL firewall workers. If no CoreXL SND's and CoreXL firewall workers are overloaded, nothing happens (picture 1).

Now, let's assume the CoreXL SNDs are overloaded (picture 2), a mathematical formula is used to calculate that a further CoreXL SND is added. In this case a CoreXL firewall worker 5 will not get any new connections (picture 3) and the connections are distributed to another CoreXL firewall worker for example to the CoreXL firewall worker 4. If there are no more connections running through this CoreXL firewall worker on core two, the core will be used for a new CoreXL SND instance (picture 4) . Now our appliance has three SND's and 5 CoreXL firewall workers.

It also works the other way round.

Picture 1 - nothing overloaded


Picture 2 - SND's overloaded

Picture 3 - CoreXL firewall worker stops the processing and distributes the connections.

Picture 4 - new SND is added

The Dynamic Split Daemon (dsd) has three stages in each iteration

1) Examine the current CPU utilization.
2) Decide if and what changes to make based on the current CPU utilization.
3) If needed, change the current CoreXL configuration

CLI Commands

In ClusterXL, you must configure all the Cluster Members in the same way. The dynamic_split command controls the Dynamic Split of CoreXL Firewall and SND instances on the local Security Gateway, or ClusterXL Member.

For more information, see R80.40 Performance Tuning Administration Guide - Chapter CoreXL.

Run these commands in the Expert mode

# dynamic_split

                            -o disable                 -> Disables the CoreXL Dynamic Split. Requires a reboot.
                            -o enable                  -> Enables the CoreXL Dynamic Split. Requires a reboot
                            -o start                      -> Starts the CoreXL Dynamic Split after it was stopped.
                            -o stop                       -> Stops the CoreXL Dynamic Split. This change survive the reboot.
                            -p                                -> Show status

[HeikoAnkenbrand]

A small update of the article with pictures.

[rolf]

Nice info!

[HeikoAnkenbrand]

Update: CLI CommandsCLI CommandsCLI Commands

[phlrnnr]

Is this enabled by default in R80.40?  Or does it have to be turned on?

[Ilya_Yusupov]

@phlrnnr  - It have to be turned on.

[HeikoAnkenbrand]

I did a cluster update to R80.40 today and have it enabled on with 16 core.

Unfortunately I cannot test it, because the cores only had a utilisation of about 10%:-)

In ClusterXL, you must configure all the Cluster Members in the same way. The dynamic_split command controls the Dynamic Split of CoreXL Firewall and SND instances on the local Security Gateway, or ClusterXL Member.

For more information, see R80.40 Performance Tuning Administration Guide - Chapter CoreXL.

Run these commands in the expert mode

# dynamic_split

                            -o disable                 -> Disables the CoreXL Dynamic Split. Requires a reboot.
                            -o enable                  -> Enables the CoreXL Dynamic Split. Requires a reboot
                            -o start                      -> Starts the CoreXL Dynamic Split after it was stopped. This change survives the reboot-
                            -o stop                       -> Stops the CoreXL Dynamic Split. This change does not survive the reboot.

                            -p                                -> Show status

[HeikoAnkenbrand]

I added that to the original article.

[Florian_Winterb]

What are the correct steps?

first -> enable

second -> start

[Harry_Morgan]

If this function is activated for r80.40 with 8 cores by default?

[HeikoAnkenbrand]

Yes, it is enabled with 8 and more cores by default.

[HristoGrigorov]

Any study on how efficient actually is this CoreXL split ? Also, do you know how often is current load evaluated and re-assignment made ?

[Pablo_Montega]

I'm using it on a 16 core system. I don't see any redistribution of cores.

[Dorit_Dor]

Did you turned it on (As mentioned above It have to be turned on).

The function is off by default (initially). It enables us to get wide production exposure before exposing everyone to the new functionality

[Jerry]

I'm using it on 24-core 15600 CXL and it makes massive difference only when SXL has more than 90% of rates so most of your SecureXL traffic hits proper templates 😛 will post something soon how this performs but I see a nice way of self-disti within CXL/SND on that ClusterXL.

[Jerry]

here is a bit I'm about to turn on about, just waiting for "reboot time" 🙂

[Expert@cp:0]# uname -a
Linux cp 3.10.0-957.21.3cpx86_64 #1 SMP Mon Jan 6 17:24:28 IST 2020 x86_64 x86_64 x86_64 GNU/Linux
[Expert@cp:0]# dynamic_split -P
P is not a valid option
Usage: enable or disable, stop or start [-o enable|disable|stop|start]
print status [-p]

[Expert@cp:0]# dynamic_split -p
Dynamic Split is currently off
ALPHA: 10
EMERGENCY_CPU_HANDLING_THRESHOLD: 40

[Jerry]

sequence of events though:

[Expert@cp:0]# cat /opt/CPsuite-R80.40/fw1/log/dynamic_split.elg
[Sat Apr 18 08:34:12 BST 2020] Dynamic Split is currently off ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:36:58 BST 2020] Dynamic Split is currently off ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:51:18 BST 2020] spreading queues
[Sat Apr 18 08:51:18 BST 2020] sorted cpus aquired
[Sat Apr 18 08:51:24 BST 2020] ON
[Sat Apr 18 08:56:16 BST 2020] Dynamic Split is currently on ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:57:37 BST 2020] Dynamic Split is currently on ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:57:58 BST 2020] OFF due to disablement
[Sat Apr 18 08:57:58 BST 2020] weights reset
[Sat Apr 18 08:57:58 BST 2020] insts started
[Sat Apr 18 08:57:58 BST 2020] insts affined
[Sat Apr 18 08:58:02 BST 2020] snds reset
[Sat Apr 18 08:58:02 BST 2020] state file removed
[Sat Apr 18 09:06:30 BST 2020] starting
[Sat Apr 18 09:06:30 BST 2020] ON following "-o start"
[Sat Apr 18 09:11:59 BST 2020] spreading queues
[Sat Apr 18 09:11:59 BST 2020] sorted cpus aquired
[Sat Apr 18 09:12:04 BST 2020] ON

+ following CCC:

[Executing:]# fw ctl affinity -l -a
Kernel fw_0: CPU 23
Kernel fw_1: CPU 11
Kernel fw_2: CPU 22
Kernel fw_3: CPU 10
Kernel fw_4: CPU 21
Kernel fw_5: CPU 9
Kernel fw_6: CPU 20
Kernel fw_7: CPU 8
Kernel fw_8: CPU 19
Kernel fw_9: CPU 7
Kernel fw_10: CPU 18
Kernel fw_11: CPU 6
Kernel fw_12: CPU 17
Kernel fw_13: CPU 5
Kernel fw_14: CPU 16
Kernel fw_15: CPU 4
Kernel fw_16: CPU 15
Kernel fw_17: CPU 3
Kernel fw_18: CPU 14
Kernel fw_19: CPU 2
Interface eth1-01: has multi queue enabled     *** 10G SFP+
Interface eth1-02: has multi queue enabled    *** 10G SFP+

🙂 any thoughts though?

Cheers mates!

Jerry

[Silvia_Day]

Nice info!

[_Val_]

Hi @HeikoAnkenbrand, a smal correction:

You say, -o stop command does not survive reboot.

The Admin guide says otherwise.

[HeikoAnkenbrand]

Was different in R80.40 EA and the presentations in Israel.

Thanks, I'll change that.

[HeikoAnkenbrand]

done 

[HeikoAnkenbrand]

Add:

• CoreXL Dynamic Split does not support:

  • Check Point Appliances with less than 8 CPU cores.

  • Check Point Appliances that run in VSX mode (regardless of the number of CPU cores).

  • Open Servers or Virtual Machines.

  • Security Gateway (or Cluster Members) with Bridge interfaces.

[Jerry]

all matches this SK though:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[_Val_]

Exactly right. We have also just had a TechTalk about the feature.

[Jerry]

we've had the Tech-Talk yesterday too 😛 with Gay and Amit 😛
and that was on particular 24-core SG which apparently does not have enough "load" in order to show-off how SND is "behaving" 🙂
but at least we've seen it all working as expected with multi-Q replaced by DS. More by enclosed in the next post.

[Jerry]

[eitan_tanami]

Will dynamic split also be released for open servers in the future.

[Chen_Muchtar]

It is currently supported only on Check Point appliances. Under evaluation per necessity and technical implications as to open servers / cloud

[phlrnnr]

Make sure you go to at least Jumbo 77 on R80.40 to fix some bugs in dynamic_split:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[AmitShmuel]

As per offline Qs with this regards, I’d like to clarify the following; While the stopped fw worker is not getting any new connections, it continues to handle its existing ones on the core which it is being moved to, also, once the fw worker is being moved, the free core immediately starts to act as an SND.