R80.x Performance Tuning Tip – SMT (Hyper Threading)

[HeikoAnkenbrand]

SMT (Hyper Threading)

Hyper Threading Technology is a form of Simultaneous Multithreading Technology (SMT) introduced by Intel. Architecturally, a processor with Hyper-Threading technology consists of two logical processors per core, each of which has its own processor architectural state.Each logical processor can be individually halted, interrupted or directed to execute a specified thread, independently from the other logical processor sharing the same physical core. Unlike a traditional dual-processor configuration that uses two separate physical processors, the logical processors in a hyper-threaded core share the execution resources. These resources include the execution engine, caches, and system bus interface; the sharing of resources allows two logical processors to work with each other more efficiently, and allows a logical processor to borrow resources from a stalled logical core (assuming both logical cores are associated with the same physical core).

A processor stalls when it is waiting for data it has sent for so it can finish processing the present thread. The degree of benefit seen when using a hyper-threaded or multi core processor depends on the needs of the software, and how well it and the firewall.

Hyper-Threading works by duplicating certain sections of the processor - those that store the architectural state - but not duplicating the main execution resources. This allows a hyper-threading processor to appear as the usual "physical" processor and an extra "logical" processor to the firewall.

The number of concurrent threads can be decided by the chip designers. Two concurrent threads per CPU core are common. Because it is really an efficiency technique that inevitably increases conflict on shared resources, measuring or agreeing on its effectiveness can be difficult.

Chapter

More interesting articles:

- R80.x Architecture and Performance Tuning - Link Collection
- Article list (Heiko Ankenbrand)

Preview to Intel Architecture

The following statement is also often discussed on the Internet:

SMT can increase message rate for multi process applications by having more logical cores. This increases the latency of a single process due to lower frequency of a single logical core when hyper-threading is enabled. This means interrupt processing of the NICs will be slower, load will be higher and packet rate will decrease. I think that's why Check Point doesn't recommend SMT in pure firewall and VPN mode. From my point of view, it only accelerates software balades. Therefore I use it if necessary, if many blades are activated. I'd like to discuss that with Check Point.

Small example with basic viewing:

This presentation is very simplified and should illustrate the issues. If SMT channel 2 uses all core resources with I/O operations, channel 1 must wait for the core resources. This can reduce the performance with enabled SMT. The same effect can occur with multi-queue and enabled SMT. The problem can be fixed by adjusting the Check Point affinity or disable SMT. What we see here, many Intel architecture issues can affect SMT and therefore the firewall performance.

Check Point and SMT

SMT is a feature that is supported on Check Point appliances running Gaia OS. When enabled, SMT doubles the number of logical CPUs on the security gateway, which enhances physical processor utilization. When SMT is disabled, the number of logical CPUs equals the number of physical cores. SMT improves performance up to 30% on software blades such as IPS, Application & URL Filtering and Threat Prevention by increasing the number of CoreXL FW instances based on the number of logical CPUs.

Turning on SMT can have some side effects in terms of multi-queue and affinity. After turning on SMT the affinity should normally be adjusted.

There are also some cases in which SMT should not be used:

•         if only firewall/VPN blades are used
•         if these blades are enabled:
  •        DLP
  •        Anti Virus

•         if these features are enabled
  •        Using services with resources in firewall policy
•         use hide NAT extensively -> Refer to sk88160 - Dynamic NAT port allocation feature to mitigate this limitation.
•         more see SK93000

Following information must also be observed before turning on SMT:

•         Changing SMT state requires reboot.
•   For supported appliance refer to "Supported Appliances".
•         If you have two ports on any of the aforementioned appliances to handle most of the traffic, it is recommended to enable the multi-queue feature to increase SMT performance.
•         For R77.30 versions only!
    If the VPN blade is enabled and a significant amount of IPSec VPN traffic passes through the security gateway, then follow the steps in SK93000.
•         For the best performance, it is recommended that user space processes not be affined to the same physical CPU core as the secure network distributors (SNDs), but rather that they be 
    affined to the firewall workers CPUs (more see SK93000, SK98737).
•         If necessary, the affine has to be adjusted here (more see SK93000).
•        Enabling SMT will load additional CoreXL FW instances. These instances consume memory and the maximal connection capacity may decrease by up to 10%.
   (info from Check Point Ofir S. )
•         more limitations see SK93000

Supported configurations for SMT:

•   only on Check Point appliances
•   only on security gateways running Gaia OS with 64-bit kernel
•   supported for R77 release and later
  
• SMT (HT) on open servers beginning from R80.40 with kernel 3.10
  (Reference : Does R80.40 support HP DL380 G10)

New with 3.10 kernel (R80.20, R80.30, R80.40):

With new kernel 3.10 for R80.20 ,R80.30 and R80.40 Check Point aligned with the industry and now HT is set and controlled by the BIOS. Therefore R80.20 and above Security Management, R80.20 and above Security Gateway with 3.10 kernel and next versions will have SMT on by default provided that the BIOS has it enabled.

Tip!

On some open servers SMT (HT) must be disabled in the BIOS for the gateway installation. This is documented in the HCL for Server see link HCL.

For example for "HP ProLitan DL360 Gen9" refert to SK Required steps before installing Gaia OS on HP ProLiant Gen9 servers.

Supported Appliances

 Attention!

SMT is supported only on following Check Point appliances not on Open Server R80.10 - R80.30. SMT on open servers beginning from R80.40 with kernel 3.10.

Appliance	Comment
3100 3200	No hardware support for SMT.
5100 5200 5400 5600	No hardware support for SMT.
5800 5900	Is already shipped with enabled SMT feature in the BIOS. SMT is recommended with all blades. Installation of the hotfix from sk109772 - R77.30 NGTP, NGTX and HTTPS Inspection performance and memory consumption optimization. Enabling of SMT feature in 'cpconfig' (refer to "Enable SMT" section).
12400	Requires 8 GB of RAM. Refer to these solutions: sk68700 for instructions for "12400 Appliance Installing and Removing Memory" sk71001 for instructions on how to upgrade to 64-bit Gaia computers Requires enabling of SMT feature both in the BIOS (1) and in 'cpconfig' (refer to "Enable SMT" section).
12600	Requires enabling of SMT feature both in the BIOS (1) and in 'cpconfig' (refer to "Enable SMT" section).
13500 13800	Is already shipped with enabled SMT feature in the BIOS. Requires enabling of SMT feature only in 'cpconfig' (refer to "Enable SMT" section).
15400 15600	Is already shipped with enabled SMT feature in the BIOS. SMT is recommended with all blades. Installation of the hotfix from sk109772 - R77.30 NGTP, NGTX and HTTPS Inspection performance and memory consumption optimization. Enabling of SMT feature in 'cpconfig' (refer to "Enable SMT" section).
21400 21600 21700	SAM Acceleration card is not supported with SMT Requires enabling of SMT feature both in the BIOS (1) and in 'cpconfig' (refer to "Enable SMT" section).
21800	Is already shipped with enabled SMT feature in the BIOS. Requires enabling of SMT feature only in 'cpconfig' (refer to "Enable SMT" section).
23500 23800	Is already shipped with enabled SMT feature in the BIOS. SMT is recommended with all blades. Installation of the hotfix from sk109772 - R77.30 NGTP, NGTX and HTTPS Inspection performance and memory consumption optimization. Enabling of SMT feature in 'cpconfig' (refer to "Enable SMT" section).
23900	Hyper Threading is hard-coded to be disabled on R77.30 and R80.10, with no impact on performance
TE250X TE1000X TE2000X	Is already shipped with enabled SMT feature in the BIOS. Requires enabling of SMT feature only in 'cpconfig' (refer to "Enable SMT" section). On these appliances, SMT is recommended with all blades.
40K  60K	SGM220      no SMT support SGM200T    no SMT support SGM260      SMT support  (20 physical cores / 40 with enabled SMT) SGM400      SMT support  (28 physical cores / 56 with enabled SMT)

(1) Note: To enable SMT in the BIOS, contact Check Point Support or contact Check Point Professional Services to get confirmation and approval beforehand.

Check if SMT is activated

 Check if SMT is activated and check current SMT status on security gateway:

# cat /proc/smt_status

Either SMT is not supported on this machine or SMT is disabled in the BIOS.

SMT is enabled in the BIOS, but disabled in 'cpconfig'

SMT is enabled in BIOS and in 'cpconfig'

If SMT is activated this command shows the nummbers of CPUs, cores and SMTs:

# grep -E "cpu cores|siblings|physical id" /proc/cpuinfo | xargs -n 11 echo |sort |uniq

Enable SMT

1) Check the number of cores

# fw ctl multik stat

2) Check the number of cores in the gateway license.
# cplic print

3) Enable SMT in the BIOS + Reboot

4) Enable SMT in Check Point software and reboot the gateway.

Attention!
  1) If multi-queue and affinity are not adjusted or used, this can lead to performance problems in combination with SMT.

  2) Before enabling SMT, follow the instructions in SK93000 to verify that the Security Gateway can support SMT safely and check if there is enough memory available for the FW_Worker.

# cpconfig

•        Choose 'Configure Hyper-Threading'
•        Select 'yes'

# reboot

5) Check the number of CoreXL FW instances

# fw ctl multik stat

If the number of CoreXL FW instances did not increase automatically, then configure the CoreXL.

# cpconfig

•         Choose 'Configure Check Point CoreXL'
•         Set the desired number of CoreXL FW instances
    (for example 3 CoreXL FW instances)

Attention!
With a ClusterXL, the core number must be the same on all gateways. Otherwise ClusterXL problems will occur.

# reboot

6) Check again the number of CoreXL FW instances

# fw ctl multik stat

Disable SMT

1) Disable SMT in Check Point software and reboot the gateway.

# cpconfig

•         Choose 'Configure Hyper-Threading'
•         Select 'No'

# reboot

2) Check the number of CoreXL FW instances

# fw ctl multik stat

Q&A

Q: Which appliances were tested with SMT enabled.

A: 5800 / 5900 / 15400 / 15600 / 23500 / 23800 / TE250X / TE1000X / TE2000X

Q: I have an Open Server. Can I enable SMT in the BIOS and use it?

A: SMT is supported only on Check Point appliances.

Q: Is any degradation expected when enabling SMT?
A: Enabling SMT will load additional CoreXL FW instances. These instances consume memory and the maximal connection capacity may decrease by up to 10%.

Q: On which appliances is SMT enabled by default in the BIOS?

A: Only on 13500 / 13800 / 15400 / 15600 / 21800 / 23500 / 23800 / TE250X / TE1000X / TE2000X appliances

Q: What to do if multi-queue is enabled?

A: If using Multi-Queue, once the final CoreXL split has been set be sure to run cpmq reconfigure and reboot again, this will help ensure the new allocation of SND/IRQ cores are properly deployed for SoftIRQ processing on the Multi-Queue-enabled interfaces.

References

SMT (HyperThreading) Feature Guide
Best Practices - Security Gateway Performance
ATRG: CoreXL 
Dynamic NAT port allocation feature 

[TomTom]

Is it possible to enable SMT on R80.40 open Server?

[PhoneBoy]

It will be once we add support for Open Server gateways in the first R80.40 JHF.

[Kaspars_Zibarts]

I might have missed that bit but you cannot turn off SMT on appliances delivered with R80.30 3.10 kernel. There's no cpconfig option and you would need to contact TAC to get BIOS password to disable it. Learnt it hard way from our 26000

[HeikoAnkenbrand]

Hi @Kaspars_Zibarts 

With new kernel 3.10 for R80.20 ,R80.30 and R80.40 Check Point aligned with the industry and now HT is set and controlled by the BIOS. Therefore R80.20 and above Security Management, R80.20 and above Security Gateway with 3.10 kernel and next versions will have SMT on by default provided that the BIOS has it enabled.

[Herschel_Liang]

I have several questions about it. If set up a cluster, which steps should we operate in order to get the least downtime? What will happen after I set up a secondary member after enabling SMT? Will it impact the network after I set up a secondary member after enabling SMT? What HA status will it be in the cluster? Can I failover through remote SSH after one member enabled SMT?

[Marcel_Gramalla]

I think you should open a new thread if you have such a question but I will try to answer anyway. If you enable SMT you will have a different core count for one member so the status will be Active Attention. You can failover without any issue but I think sessions that are on cores that are not existent on the other member will be lost as the connection table cannot be synchronized for that. 

So I would start enabling SMT on the standby member --> reboot and then do a failover to the other member. And then repeat the process. You will end up with zero downtime but a few open connections might be lost as I said.

[Herschel_Liang]

After standby member enabled SMT and rebooted, will the cluster become A/A but not A/S handle the traffic?

[Kaspars_Zibarts]

I agree with @Marcel_Gramalla - pls open a new thread for this. There is a lot you need to think through. I.e.

• is it a VSX or a regular gateway?
• how loaded are your FWKs?
• you need take into account that SMT cores will be less powerful so you need to do it the maintenance window
• then you most likely will have to increase FWK core count for the same reason

So there is a quite a bit of planning to be done before you go ahead. We did it on our VSX quite a while ago, so I would have to check my notes if there was something more. Have a vague memory that we had clustering issues during split stage, but I might be wrong