Tip of the Week - VPN Performance Best Practices

[_Val_]

If you are heavily using VPNs, both IPsec and SSL based, you may need to tune your Security Gateways for better performance. This SecureKnowledge article provides general performance guidelines for working with VPN.

[Timothy_Hall]

That's a great SK Val, this other SK is a bit lengthy but provides great insights into the inner workings of Check Point VPNs and is highly recommended:

sk104760: ATRG: VPN Core

[Aidan_Luby]

I've asked this more than once on CheckMates and also to my Account Engineer and I'd like to ask again. Why is it the 6000 series firewalls do not support AES-NI when the CPU does? I will probably be deploying 6500's soon and that was one of the main reasons I wanted them; the CPU's support AES-NI.

We have many VPN tunnels and would benefit from this greatly. We also use HTTPS Inspection as well.

[_Val_]

@Aidan_Luby , according to this SK https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... it is actually supported. Who says it is not?

[Aidan_Luby]

Because some commands I try to confirm it seem to indicate it might be activated but others do not. For example:

#dmesg | grep "AES-NI"
[fw4_0];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_0];VPN-1: AES-NI is supported on this hardware
[fw4_1];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_1];VPN-1: AES-NI is supported on this hardware
[fw4_2];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_2];VPN-1: AES-NI is supported on this hardware
[fw4_3];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_3];VPN-1: AES-NI is supported on this hardware
[fw4_4];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_4];VPN-1: AES-NI is supported on this hardware
[fw4_5];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_5];VPN-1: AES-NI is supported on this hardware
#grep -m1 -o aes /proc/cpuinfo

#

I wish there was something more cut and dry than "AES-NI is supported on this hardware". Something which says it's supported then another message which says it's activated would be preferred.

[Timothy_Hall]

OK so the AES-NI question on the 6000 series has been bugging me ever since this thread:

https://community.checkpoint.com/t5/Enterprise-Appliances-and-Gaia/New-6000-series-appliances/m-p/33...

sk105119: Best Practices - VPN Performance and Intel's specs clearly state that the 6000 series processors support AES-NI, but users have reported that the processor extension flag "aes" is not showing up in the output of cat /proc/cpuinfo on the 6000 series units, leading to speculation that it is disabled in the BIOS or perhaps not supported by Gaia.

As fate would have it, earlier this week I was setting up a 1Gbps Intranet VPN for a customer between two clusters of 6500 units.  While we were waiting for our cutover window to start I poked around a bit.  Version of both clusters was R80.30 kernel 2.6.18, 4/4 CoreXL split w/ Multi-Queue due to a large amount of fully-accelerated traffic.  When we enabled the IPSec VPN blade for the first time and installed policy, this immediately showed up in dmesg:

[fw4_0];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_0];VPN-1: AES-NI is supported on this hardware
[fw4_1];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_1];VPN-1: AES-NI is supported on this hardware
[fw4_2];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_2];VPN-1: AES-NI is supported on this hardware
[fw4_3];VPN-1: AES-NI is allowed on this machine. Testing hardware support
[fw4_3];VPN-1: AES-NI is supported on this hardware

This verbiage would seem to suggest that a test operation invoking AES-NI was attempted and succeeded, not just that AES-NI was being reported as a valid processor extension.  As others have observed, I saw that running cat /proc/cpuinfo does not show aes listed on a 6500 (output trimmed for brevity):

model name : Intel(R) Core(TM) i7-4790S CPU @ 3.20GHz

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx pdpe1gb rdtscp lm constant_tsc pni monitor ds_cpl vmx smx est tm2 cx16 xtpr popcnt lahf_lm altm ovcr8

There don't seem to be any counters or kernel variables that indicate whether AES-NI is actually being used that I could find, so I decided to take a different tack:

[Expert@GW2:0]# cpopenssl speed aes-256-cbc
Doing aes-256 cbc for 3s on 16 size blocks: 14084079 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 64 size blocks: 3811710 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 974955 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 1024 size blocks: 245997 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 8192 size blocks: 30781 aes-256 cbc's in 3.00s
OpenSSL 1.0.2r 26 Feb 2019
built on: reproducible build, date unspecified
options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include -I/local_ckp/xm/makefile/heat_main/B993000015/include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DDEFINE_IN_OPENSSL_ONLY -Wa,--noexecstack -DL_ENDIAN -g -O3 -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type               16 bytes    64 bytes    256 bytes  1024 bytes 8192 bytes
aes-256 cbc 75115.09k 81316.48k 83196.16k 83966.98k 84052.65k

The AES-256 speed test results above did not attempt to utilize any processor extensions, just raw CPU power.  Next I invoked the -evp option, which instructs openssl to utilize the EVP API to invoke all possible processor extensions in an attempt to make the AES operations run faster:

[Expert@GW2:0]# cpopenssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 89992230 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 23707621 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 6005383 aes-256-cbc's in 2.99s
Doing aes-256-cbc for 3s on 1024 size blocks: 1506342 aes-256-cbc's in 2.99s
Doing aes-256-cbc for 3s on 8192 size blocks: 188446 aes-256-cbc's in 3.00s
OpenSSL 1.0.2r 26 Feb 2019
built on: reproducible build, date unspecified
options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include -I/local_ckp/xm/makefile/heat_main/B993000015/include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DDEFINE_IN_OPENSSL_ONLY -Wa,--noexecstack -DL_ENDIAN -g -O3 -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type               16 bytes       64 bytes      256 bytes    1024 bytes  8192 bytes
aes-256-cbc 479958.56k 505762.58k 514173.26k 515884.35k 514583.21k

Yowza, a 6.1X increase in AES speed, which falls right within the expected 4-10X performance boost provided by AES-NI.  Now it could be argued that some processor extensions other than AES-NI can assist portions of the AES operation (mainly sse and avx but perhaps some others) but based on the Intel document below I don't see how these other extensions could provide more than about 1.5X of boost, maybe even up to 2.0X in ideal conditions:

https://software.intel.com/en-us/articles/improving-openssl-performance#_Toc416943486

So assuming that the cpopenssl tool results can be trusted and I'm not misinterpreting them, AES-NI is definitely present and being actively utilized on the 6000 series.  There is simply NO WAY we could get such a hefty 6.1X performance boost unless AES-NI is present and working.

The next question of course is why aes is not showing up in the output of cat /etc/procinfo.  Well as it turns out that according to Intel's site the Intel(R) Core(TM) i7-4790S CPU @ 3.20GHz processor supports another extension called AVX, but it isn't listed in the output either.  So I can only conclude that the processor extensions are not being properly displayed, as the Check Point code certainly seems to think AES-NI is there, and based on the speed results openssl does too. 

I'm tagging @HeikoAnkenbrand to review my findings as he has a bit more knowledge of the processor hardware architectures than I do.

[Aidan_Luby]

You've made my day @Timothy_Hall. I'm constantly blown away by the amount of time you put into helping everyone in this community and I really appreciate the time you've put into this particular issue.

I feel much better about my impending 6500 Cluster installation now.

[HeikoAnkenbrand]

Hi @Aidan_Luby 

Thanks @Timothy_Hall. I have provided the links to the articles below.

You can found more informations to AES-NI and Intel hardware here in my articles:

- R80.x - Performance Tuning Tip - AES-NI

- R80.x - Performance Tuning Tip - Intel Hardware

[Michael_Thompso]

Hi Timothy and all, I ran the following command (cpopenssl speed -evp aes-256-cbc) on a 23800 series. I did see a drastic improvement versus running without the -evp flag. However, I would expect my results to better than the 6500s. I get the following:

[Expert@nycvpn1:0]# cpopenssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 70302382 aes-256-cbc's in 2.99s
Doing aes-256-cbc for 3s on 64 size blocks: 18504937 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 4688988 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 1176151 aes-256-cbc's in 2.99s
Doing aes-256-cbc for 3s on 8192 size blocks: 147058 aes-256-cbc's in 2.99s
OpenSSL 1.0.2r 26 Feb 2019
built on: reproducible build, date unspecified
options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx)
compiler: gcc -I. -I.. -I../include -I/local_ckp/xm/makefile/gogo_heat_188_main/B993000002/include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DDEFINE_IN_OPENSSL_ONLY -Wa,--noexecstack -DL_ENDIAN -g -O3 -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 376200.04k 394771.99k 400126.98k 402802.22k 402909.41k

Is that not the case? 

[Timothy_Hall]

The 23800 has a slightly older processor (Xeon E5-2680v3) vs. the 6500 (i7-4790S).  While the 23800 has many more cores, the 6500 is actually faster per core, and I'm pretty sure that cpopenssl test uses only one core.  See here:

https://cpu.userbenchmark.com/Compare/Intel-Xeon-E5-2680-v2-vs-Intel-Core-i7-4790S/m17083vsm13424

[Michael_Thompso]

Thanks! 

[Timothy_Hall]

As far as AES-NI support being present but not showing up as being present in various ways, the mystery has been solved.  See here: sk170779: AES-NI commands no longer work in R80.40

[Aidan_Luby]

You really clung onto this one and didn't let go @Timothy_Hall 😀

I think the document could be updated to list all the versions that those commands fail for. I believe at least it would be R80.30 and R80.20 as R80.40 wasn't out when I first noticed this issue.

[HristoGrigorov]

A but funny SK.. It claims 'dmesg' and 'grep' commands to have been removed while in fact they are there, just no output as mentioned. I personally did not understood what the problem is and how it is fixed in R81 T13. 

[_Val_]

@HristoGrigorov  It is badly worded. I have asked to fix.

[DR_74]

Hi,

On my side, I am running a 5800 cluster with R80.30 and can't see any trace of AES-NI.

From what I read 58xx appliances should be OK (https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...)

Either dmesg command or cat /proc/cpuinfo do not show any AES-NI

cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 71
model name : Intel(R) Xeon(R) CPU E3-1285L v4 @ 3.40GHz
stepping : 1
cpu MHz : 3392.402
cache size : 6144 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 20
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx pdpe1gb rdtscp lm constant_tsc pni monitor ds_cpl vmx smx est tm2 cx16 xtpr popcnt lahf_lm altmovcr8 misalignsse
bogomips : 6794.47
clflush size : 64
cache_alignment : 64
address sizes : 39 bits physical, 48 bits virtual
power management:

[Timothy_Hall]

It is there and working, try fw ctl get int AESNI_is_supported

sk170779: AES-NI commands no longer work in R80.40 and R81

[DR_74]

@TimhUnfortunatly, this command is not here with R80.30

On the other side when checking the Proc datasheet, we can imagine that AES-NI is there

https://ark.intel.com/content/www/us/en/ark/products/88043/intel-xeon-processor-e31285l-v4-6m-cache-...

[Timothy_Hall]

@DR_74 see my response dated "2019-10-09 09:43 AM" further up in this thread, in R80.30 you'll have to run a series of benchmarks (both utilizing processor extensions and not) and compare the results to verify that AES-NI is active.  According to R&D via @PhoneBoy:

Small addition to this thread from R&D:
AES-NI is supported and enabled out-of-the-box for 2.6.18 kernels.
In 3.10 kernels it’s supported and enabled out-of-the-box since R80.30 GA.