SSL inspection performance question on 15K cluster.

[Jeffrey_Ball]

We are running a 15400 cluster with 2 members, each with 64GB RAM.  I am trying to determine the impact of enabling SSL Inspection in terms of performance (CPU/Throughput/Load) and impact (Latency/Delay introduced to users).

I have read all of the datasheets, and searched here, but cant find any numbers on throughput or load, only that they are "Optimized for inspecting SSL-encrypted traffic"

We are currently running R77.30 on the Gateways, R80.10 on the management, and will be upgrading the gateways to R80.10 next month

[Danny · Accepted solution]

HTTPS Inspection - Best Practices (sk108202) and the HTTPS Inspection FAQ (sk65123) detail the performance impact one has to be aware of. Also make sure you have the latest available Jumbo Hotfix R77.30 | R80.10 installed. We're using HTTPS Inspection for almost all our customers. Check Point has developed various HTTPS Inspection performance boosts over the years. Most of the issues related to HTTPS Inspection as described in Check Points Knowledgebase are fixed with the latest Jumbo Hotfixes.

HTTPS Inspection requires the Security Gateway to perform extra SSL work:

• SSL handshake with the secure web site and with the client browser.
• Decrypt & re-encrypt all SSL traffic, to be able to inspect it.

This has some performance impact on SSL capacity and latency, but in normal situations the end user should not be aware of it.

HTTPS Inspection creates additional load on Security Gateway's CPU due to these reasons:

• SSL termination, encrypt/decrypt and active TCP termination that consume CPU resources (Note: The SSL handshake rate was significantly improved in R77.30 - refer to sk103081 and to sk104717).
• Additional traffic is inspected by the security blades.

It is possible to approximate the effect of HTTPS Inspection activation under the following disclaimers:

1. Representing a typical, outbound configuration (low or none inbound HTTPS Inspection traffic) with 36% HTTPS.
2. Using R77.30 with either NGTP (Firewall, IPS, Application Control, URL Filtering, Anti-Virus, and Anti-Bot), or NGFW (Firewall, IPS and Application Control) software blades for inspection.
3. Data Center scenario requires specific modeling.

The rational is that under the disclaimers written above, the impact on required Security Power (SPU) is 60% to 100% higher depending on the enabled software blades (the more blades are already enabled, the smaller the additional impact of HTTPS Inspection will be).

Therefore, when enabling HTTPS Inspection in an existing configuration, the CPU utilization on Security Gateway is expected to increase:

• by factor of 1.6 for NGTP configuration
• by factor of 2 for NGFW configuration

For Check Point Partners: refer to sk108757 - How to estimate the performance impact of HTTPS Inspection using the Appliance Sizing Too....

HTTPS Inspection Enhancements in R77.30 and above

Some HTTPS sites do not load when HTTPS Inspection is enabled, if TLS 1.2 with ECDHE cipher is used

Specific HTTPS sites that use ECDHE ciphers are not accessible when HTTPS Inspection is enabled

How to debug WSTLSD daemon

[Luis_Miguel_Mig]

Is it not the appliance sizing tool available any more?

sk108757 - How to estimate the performance impact of HTTPS Inspection using the Appliance Sizing Too....

I was trying to find out the impact HTTPS inspection could have.

I have a gateway with 20 Mbps of HTTPS traffic. Once I have HTTPS inspection I would  enable Application Control, URL filtering, IPS, Anti-Virus, Anti-Bot, Threat Prevention and Threat Extraction on top of the firewall funcionality.

[Hugo_vd_Kooij]

There is slightly more to it with R77.30.

If you enable HTTPS inspection you will force more traffic to the Application blade then you might expect.

So I tend to keep at least a factor of 2 as CPU impact on the fw workers.

And there is a long list of items to check with R77.30 as it depends on various settings and the Exact Jumbo Hotfix installed. We tend to move to 28x versions if a customer wants HTTPS inspectionand then go over the load of variables you need to tweak. But you will get complaints from your users about sites being slower, sites not working correctly, ......

Some of these complaits will be real issues. To the best of my knowledge there ae still outstanding issue that either require a custom patch or ar not yet solved.

Some of these may result from silly things like a Dutch certificate provider throwing in a SERIALNUMBER string into a Subject filed of the certificate.

[Danny]

That's correct. Depending on the type and number of HTTPS services and end users there will be a growing list of workarounds and HTTPS Inspection bypasses required. Escpecially if you encounter a site with certificate pinning. At least I don't remember anything that required a custom patch. If services like Skype and Co. cause any trouble, you can easily create a workaround (sk114419) by defining network objects to represent ranges on IP addresses used by the clients to bypass those.

[Timothy_Hall]

Agree that factor of 2 is a relatively safe assumption for performance impact of HTTPS Inspection.

A side effect of enabling HTTPS Inspection from a tuning perspective is that any traffic subject to it will be handled in the Firewall Path (F2F) instead of the Medium Path (PXL) where it was probably being handled previously.

If users complain about slow initial load times when first connecting to a website subject to HTTPS Inspection by the firewall (but performance to the site is fine after that), keep in mind that the initial HTTPS negotiation and key calculations are handled in process space by numerous wstlsd daemons (one spawned for and associated with each Firewall Worker Core) and a single pkxld process (R77.30+ only).  If there is heavy CPU utilization in the kernel (sy/si) across all cores (R77.30) or only the worker cores (R80.10+), these daemons might start to get starved for CPU by the kernel and cause slow initial HTTPS site loads.  Once the initial HTTPS negotiations & key calculations are complete for a site, mainline SSL encryption/decryption then takes place in the kernel via F2F.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Jeffrey_Ball]

Thanks to everyone for the feedback, this detailed information has been very helpful.  With our 15400's running 6-8%, I thing we will be fine, I will just have to be prepared with workarounds for the inevitable issues requiring bypass  as Danny points out.  Thanks again to the community!

[Kaspars_Zibarts]

Hi Jeffery, I would be very interested in hearing results! How did theory apply into practice?