Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
HeikoAnkenbrand
Champion Champion
Champion

R80.40 - Dynamic split of CoreXL

 

What is new in R80.40 EA.

A new interesting function for performance tuning has been included in R80.40. Dynamic split of CoreXL changes the assignment of  CoreXL SND's and CoreXL firewall workers automatically without reboot.

How does this magic happens?

  • Adding and removing a CoreXL firewall worker
  • Adding and removing a CoreXL SND
  • Balance between CoreXL SND and CoreXL firewall worker
  • Work in ClusterXL environments
  • A reboot is not necessary

Pre-requisites:

  • GAIA 3.10 kernel (USFW/Kernel)
  • only Check Point appliances with 8 cores or more
  • currently supported on ClusterXL HA
  • currently VSLS is a limitation
  • In the following series all models are supported: 7000, 15000, 16000, 23000, 26000, 28000
  • In the following series only the listed models are supported:
    • 5000: 5800 and 5900
    • 6000: 6500, 6700, 6800 and 6900
  • Supported versions: Check Point R80.40 with Jumbo Hotfix Take 25 and above
  • CoreXL Dynamic Split does not support:

    • Check Point Appliances that run in VSX mode (regardless of the number of CPU cores).

    • Open Servers or Virtual Machines.

    • Security Gateway (or Cluster Members) with Bridge interfaces.

How does it work?


Suppose we have two SND's and 6 CoreXL firewall workers. If no CoreXL SND's and CoreXL firewall workers are overloaded, nothing happens (picture 1).

Now, let's assume the CoreXL SNDs are overloaded (picture 2), a mathematical formula is used to calculate that a further CoreXL SND is added. In this case a CoreXL firewall worker 5 will not get any new connections (picture 3) and the connections are distributed to another CoreXL firewall worker for example to the CoreXL firewall worker 4. If there are no more connections running through this CoreXL firewall worker on core two, the core will be used for a new CoreXL SND instance (picture 4) . Now our appliance has three SND's and 5 CoreXL firewall workers.

It also works the other way round.

Picture 1 - nothing overloaded
DC1.JPG

Picture 2 - SND's overloaded
DC2.JPG

Picture 3 - CoreXL firewall worker stops the processing and distributes the connections.
DC3.JPG

 

Picture 4 - new SND is added
DC4.JPG

The Dynamic Split Daemon (dsd) has three stages in each iteration

1) Examine the current CPU utilization.
2) Decide if and what changes to make based on the current CPU utilization.
3) If needed, change the current CoreXL configuration

CLI Commands


In ClusterXL, you must configure all the Cluster Members in the same way. The dynamic_split command controls the Dynamic Split of CoreXL Firewall and SND instances on the local Security Gateway, or ClusterXL Member.

For more information, see R80.40 Performance Tuning Administration Guide - Chapter CoreXL.

Run these commands in the Expert mode

# dynamic_split

                            -o disable                 -> Disables the CoreXL Dynamic Split. Requires a reboot.
                            -o enable                  -> Enables the CoreXL Dynamic Split. Requires a reboot
                            -o start                      -> Starts the CoreXL Dynamic Split after it was stopped.
                            -o stop                       -> Stops the CoreXL Dynamic Split. This change survive the reboot.
                            -p                                -> Show status

 

➜ CCSM Elite, CCME, CCTE
30 Replies
HeikoAnkenbrand
Champion Champion
Champion

A small update of the article with pictures.

➜ CCSM Elite, CCME, CCTE
rolf
Participant

Nice info!

0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Update: CLI CommandsCLI CommandsCLI Commands

➜ CCSM Elite, CCME, CCTE
phlrnnr
Advisor

Is this enabled by default in R80.40?  Or does it have to be turned on?

0 Kudos
Ilya_Yusupov
Employee
Employee

@phlrnnr  - It have to be turned on.

HeikoAnkenbrand
Champion Champion
Champion

I did a cluster update to R80.40 today and have it enabled on with 16 core.

Unfortunately I cannot test it, because the cores only had a utilisation of about 10%:-)

In ClusterXL, you must configure all the Cluster Members in the same way. The dynamic_split command controls the Dynamic Split of CoreXL Firewall and SND instances on the local Security Gateway, or ClusterXL Member.

For more information, see R80.40 Performance Tuning Administration Guide - Chapter CoreXL.

Run these commands in the expert mode

# dynamic_split

                            -o disable                 -> Disables the CoreXL Dynamic Split. Requires a reboot.
                            -o enable                  -> Enables the CoreXL Dynamic Split. Requires a reboot
                            -o start                      -> Starts the CoreXL Dynamic Split after it was stopped. This change survives the reboot-
                            -o stop                       -> Stops the CoreXL Dynamic Split. This change does not survive the reboot.

                            -p                                -> Show status

 

➜ CCSM Elite, CCME, CCTE
HeikoAnkenbrand
Champion Champion
Champion

I added that to the original article.

➜ CCSM Elite, CCME, CCTE
Florian_Winterb
Participant

What are the correct steps?

first -> enable

second -> start

0 Kudos
Harry_Morgan
Contributor

If this function is activated for r80.40 with 8 cores by default?

0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Yes, it is enabled with 8 and more cores by default.

➜ CCSM Elite, CCME, CCTE
0 Kudos
HristoGrigorov

Any study on how efficient actually is this CoreXL split ? Also, do you know how often is current load evaluated and re-assignment made ?

Pablo_Montega
Contributor

I'm using it on a 16 core system. I don't see any redistribution of cores.

Dorit_Dor
Employee
Employee

Did you turned it on (As mentioned above It have to be turned on).

The function is off by default (initially). It enables us to get wide production exposure before exposing everyone to the new functionality

Jerry
Mentor
Mentor

I'm using it on 24-core 15600 CXL and it makes massive difference only when SXL has more than 90% of rates so most of your SecureXL traffic hits proper templates 😛 will post something soon how this performs but I see a nice way of self-disti within CXL/SND on that ClusterXL.
Jerry
Jerry
Mentor
Mentor

here is a bit I'm about to turn on about, just waiting for "reboot time" 🙂

[Expert@cp:0]# uname -a
Linux cp 3.10.0-957.21.3cpx86_64 #1 SMP Mon Jan 6 17:24:28 IST 2020 x86_64 x86_64 x86_64 GNU/Linux
[Expert@cp:0]# dynamic_split -P
P is not a valid option
Usage: enable or disable, stop or start [-o enable|disable|stop|start]
print status [-p]

[Expert@cp:0]# dynamic_split -p
Dynamic Split is currently off
ALPHA: 10
EMERGENCY_CPU_HANDLING_THRESHOLD: 40
Jerry
Jerry
Mentor
Mentor

sequence of events though:

 

[Expert@cp:0]# cat /opt/CPsuite-R80.40/fw1/log/dynamic_split.elg
[Sat Apr 18 08:34:12 BST 2020] Dynamic Split is currently off ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:36:58 BST 2020] Dynamic Split is currently off ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:51:18 BST 2020] spreading queues
[Sat Apr 18 08:51:18 BST 2020] sorted cpus aquired
[Sat Apr 18 08:51:24 BST 2020] ON
[Sat Apr 18 08:56:16 BST 2020] Dynamic Split is currently on ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:57:37 BST 2020] Dynamic Split is currently on ALPHA: 10 EMERGENCY_CPU_HANDLING_THRESHOLD: 40
[Sat Apr 18 08:57:58 BST 2020] OFF due to disablement
[Sat Apr 18 08:57:58 BST 2020] weights reset
[Sat Apr 18 08:57:58 BST 2020] insts started
[Sat Apr 18 08:57:58 BST 2020] insts affined
[Sat Apr 18 08:58:02 BST 2020] snds reset
[Sat Apr 18 08:58:02 BST 2020] state file removed
[Sat Apr 18 09:06:30 BST 2020] starting
[Sat Apr 18 09:06:30 BST 2020] ON following "-o start"
[Sat Apr 18 09:11:59 BST 2020] spreading queues
[Sat Apr 18 09:11:59 BST 2020] sorted cpus aquired
[Sat Apr 18 09:12:04 BST 2020] ON

 

+ following CCC:

 

[Executing:]# fw ctl affinity -l -a
Kernel fw_0: CPU 23
Kernel fw_1: CPU 11
Kernel fw_2: CPU 22
Kernel fw_3: CPU 10
Kernel fw_4: CPU 21
Kernel fw_5: CPU 9
Kernel fw_6: CPU 20
Kernel fw_7: CPU 8
Kernel fw_8: CPU 19
Kernel fw_9: CPU 7
Kernel fw_10: CPU 18
Kernel fw_11: CPU 6
Kernel fw_12: CPU 17
Kernel fw_13: CPU 5
Kernel fw_14: CPU 16
Kernel fw_15: CPU 4
Kernel fw_16: CPU 15
Kernel fw_17: CPU 3
Kernel fw_18: CPU 14
Kernel fw_19: CPU 2
Interface eth1-01: has multi queue enabled     *** 10G SFP+
Interface eth1-02: has multi queue enabled    *** 10G SFP+

 

🙂 any thoughts though?

 

Cheers mates!

 

Jerry

 

Jerry
0 Kudos
Silvia_Day
Contributor

Nice info!

0 Kudos
_Val_
Admin
Admin

Hi @HeikoAnkenbrand, a smal correction:

 

You say, -o stop command does not survive reboot.

Screenshot 2020-05-08 at 11.19.13.png

The Admin guide says otherwise.

Screenshot 2020-05-08 at 11.21.33.png

 

 

0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Was different in R80.40 EA and the presentations in Israel.

Thanks, I'll change that.

➜ CCSM Elite, CCME, CCTE
HeikoAnkenbrand
Champion Champion
Champion

done 

➜ CCSM Elite, CCME, CCTE
0 Kudos
HeikoAnkenbrand
Champion Champion
Champion

Add:

 

  • CoreXL Dynamic Split does not support:

    • Check Point Appliances with less than 8 CPU cores.

    • Check Point Appliances that run in VSX mode (regardless of the number of CPU cores).

    • Open Servers or Virtual Machines.

    • Security Gateway (or Cluster Members) with Bridge interfaces.

 

➜ CCSM Elite, CCME, CCTE
Jerry
Mentor
Mentor

_Val_
Admin
Admin

Exactly right. We have also just had a TechTalk about the feature.

0 Kudos
Jerry
Mentor
Mentor

we've had the Tech-Talk yesterday too 😛 with Gay and Amit 😛
and that was on particular 24-core SG which apparently does not have enough "load" in order to show-off how SND is "behaving" 🙂
but at least we've seen it all working as expected with multi-Q replaced by DS. More by enclosed in the next post.
Jerry
Jerry
Mentor
Mentor

SND.jpg

Jerry
eitan_tanami
Participant

Will dynamic split also be released for open servers in the future.

Chen_Muchtar
Employee
Employee

It is currently supported only on Check Point appliances. Under evaluation per necessity and technical implications as to open servers / cloud

phlrnnr
Advisor

Make sure you go to at least Jumbo 77 on R80.40 to fix some bugs in dynamic_split:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

ds.jpg

 

0 Kudos
AmitShmuel
Employee
Employee

As per offline Qs with this regards, I’d like to clarify the following; While the stopped fw worker is not getting any new connections, it continues to handle its existing ones on the core which it is being moved to, also, once the fw worker is being moved, the free core immediately starts to act as an SND.

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events