max performance / throughput of site2site-VPN

[Wolfgang]

Dear Checkmates,

I had a question regarding the throughput of one VPN site2site-tunnel.
We did some research with different appliances but did not get more then 900Mb/s for a single connection.

We tested 5600, 5800, 13800 and 15400, all with the same result.
SecureXL is on, all VPN trafic is fully accelerated.
Incoming and outgoing interface are on different SNDs.
We do not see more then 50% CPU utilization on both cores for the SNDs, looks like there are some credits before hitting 100%CPU.
Hardware encryption via AES-NI is on and we use the best IPSEC-parameter following sk73980.

The datasheet for an 5600 appliance shows 6,5GBbs and following sk73980 there should be 429% refer the datasheet possible.
I know this is a marketing value and only relevant if using multiple connections. But I hope there can be more possible.

Has anyone seen more then 1Gb/s throughput on a single VPN-tunnel and a single connection beetween devices behind two CheckPoint devices?

Any ideas are welcome.

regards
Wolfgang

[G_W_Albrecht]

Please consult sk105119: Best Practices - VPN Performance first.

[Wolfgang]

Thanks Günther,

we checked these sk twice but with no success.

Wolfgang

[Maarten_Sjouw]

when using 10Gb interfaces this should be achievable, if however you are using port channels / bonding, that will be your limiting factor. On the 1Gb interfaces even if you combine 5 of them in a port channel/bond, there is still a hashing mechanism that will assign the traffic to a specific port in the channel, this is where you run into the single session limit of the port channel of 1 interface.

[Wolfgang]

You’re right Maarten. A single connection will be run everytime over only one interface and is processed only on one core.

But we are using 10G interfaces. It looks like this is a limitation oft he system, maybee not the hardware but the software.

Wolfgang

[PhoneBoy]

If it's between a single source to a single destination (independent of VPN), then this is expected behavior.

We call this an "elephant flow."

[Steffen_Appel]

Is this a limitation per tunnel?

So if we are connecting to sites does it make a difference, if we choose one tunnel per GW/network/host?

[Timothy_Hall]

It is not a limitation per VPN tunnel specifically, so modifying the VPN Tunnel Sharing value will have no effect on your issue, the limitation is per individual connection (and a single VPN tunnel can carry multiple connections of course).  All the packets of a single connection (TCP or UDP) can only be handled by one core (whether it is a SND/IRQ or Firewall Worker), so in the case of an elephant flow all those packets will hit a single core and cannot be spread around to different cores to relieve the load.   Trying to spread the packets for a single elephant flow connection around to multiple cores raises the specter of out-of-order delivery, which if it happens will cause TCP to sharply curtail its send rate due to the congestion/problems it is seeing in the network flow and is an unmitigated disaster from a performance perspective.

Should this single-core limitation per connection ever be lifted, there would need to be some kind of reordering mechanism on the firewall to ensure packets are delivered in order.  However doing so would bottleneck the flow of a connection's packets down to the speed of the slowest or most congested core handling packets for that connection, as packets that have already been inspected and are ready for transmission get held/queued waiting for earlier packets to clear inspection on the slowest or most congested core.  Ugh.

[Steffen_Appel]

OK, thanks, that answers my question!

[Checkpoint_GoSe]

Do you know if this also applies to upload speeds?

Currently dealing with one site-to-site vpn where officially the speeds should be 200 down and 30 up.

Tests reveal that it's closer to 75 down and 27 up from across the tunnel. Download speeds appear to be more affected than upload.

Does this make sense?

[Timothy_Hall]

@Checkpoint_GoSe it might make sense, please provide the output of netstat -ni on your firewall to ensure the underlying network is running clean.  Also while running your speed test execute the top command and hit 1 to show individual core utilization, are any cores topping out at 100% during the upload portion of the test? 

[Magnus-Holmberg]

Is it possible to see updated information on the sk73980
I mean some of the newer versions such as R81.20 states in the improvments.
What is the base numbers in the SK based on for version?
Based on the SK itself i would assume its based on R80.30

Taken from R81.20

"

IPsec VPN

• Scalable VPN performance - 3 times faster to process simultaneous Remote Access and Site to Site VPN connections.
• Major performance and stability improvement for Remote Access VPN and Site to Site VPN that delivers a significantly greater capacity for VPN tunnels.
• Extended Security Gateway certificate validation capabilities for quicker authentication.
• Resilient VPN architecture - multi-process architecture to handle IKE negotiations in dedicated scalable daemons, providing unprecedented resiliency.

"

Or maybe highlight something in it regarding newer versions?

Regards,
Magnus

[Chris_Atkinson]

Feedback was submitted on the SK.

Whilst the enhancements sighted don't necessarily speak to performance improvements in terms of throughout it doesn't exclude their being differences between versions either.

[HeikoAnkenbrand]

Hello Wolfgang,

6,5GBbs is total throughput that an appliance can process via VPN. This applies to all summated connections. 

900Mb/s for a "elephant flow". It's a very good value. 

I think the VPN running over an internet interface. This makes runtime problems of the connection critical in the WAN. If you send a TCP packet over the VPN route, there are still runtimes in the WAN to consider. This is related to the windows sizing for TCP connections. This means that you send some packets in one direction and have to wait for the answer packet. It's always slowing you down on the WAN line and it's not a Ckeck Point problem, it's a WAN problem.

If you want to speed up a single connection "elephant flow", you need systems that optimize TCP and optimize protocols. I don't want to advertise for other manufacturers in the Check Point forum, but have a look at Riverbed Steeheadˋs. 

Intel‘s AES New Instructions AES-NI is a encryption instruction set that improves on the Advanced Encryption Standard (AES) algorithm and accelerates the encryption of data in many processor familys. Comprised of seven new instructions, AES-NI gives your environment faster, more affordable data protection and greater security.

For more informations about AES-NI see this article:

R80.x Performance Tuning Tip - AES-NI 

You may have other problems on the WAN interface that you can't control:
- fragmentation > check MTU size
- lost VPN packets > and therefore lost TCP packets and therfore TCP retransmissions

- packed in wrong sequence  > and therfore TCP retransmissions

Regards

Heiko

[Wolfgang]

Dear Heiko,

thanks for your response.

Our setup has no internet connection, we used leased lines between our gateways. No MTU issues, no fragmentation problems, no CPU 100% utilization and best suggested parameters for the IPSEC-values.

At CPX I had some interesting discussions with CheckPoints R&D guys.

Now I understand, 900Mb/s for one connection is a very good value. But with CheckPoint there is no chance to get more for this type of traffic, this is a really limitation.

With more connections and more then one VPN-tunnel we can get much more throughput. I found out in the last days, CheckPoint is one of the most valuable vendor for this kind of traffic but for the special case with only one connection we have to lock for another. We found one, another then you mentioned Heiko.

Another thing I learned..... CheckPoint is not alone with the problem of a single connection and „elephant flows“, but other vendors are not really better and most of them are much slower.

thanks for all suggestion,

Wolfgang

[Steffen_Appel]

Which alternative vendor did you finally use?

[Ave_Joe]

So in this elephant scenario would faster CPU cores allow higher throughput results? For instance Open Server may be an option getting better single core performance.

[Timothy_Hall]

@Ave_Joe  faster cores would allow better performance in your situation, as well as using AES encryption instead of 3DES which is older and much slower.  The Check Point Solutions Center does have a software feature available that can spread the processing of elephant flows across multiple cores, but I don't think it is in the main train of code yet.

[SimonD]

Do you have more information about this feature ?

SimonD