cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post
Highlighted
Wolfgang
Silver

max performance / throughput of site2site-VPN

Dear Checkmates,

I had a question regarding the throughput of one VPN site2site-tunnel.
We did some research with different appliances but did not get more then 900Mb/s for a single connection.

We tested 5600, 5800, 13800 and 15400, all with the same result.
SecureXL is on, all VPN trafic is fully accelerated.
Incoming and outgoing interface are on different SNDs.
We do not see more then 50% CPU utilization on both cores for the SNDs, looks like there are some credits before hitting 100%CPU.
Hardware encryption via AES-NI is on and we use the best IPSEC-parameter following sk73980.

The datasheet for an 5600 appliance shows 6,5GBbs and following sk73980 there should be 429% refer the datasheet possible.
I know this is a marketing value and only relevant if using multiple connections. But I hope there can be more possible.

Has anyone seen more then 1Gb/s throughput on a single VPN-tunnel and a single connection beetween devices behind two CheckPoint devices?

Any ideas are welcome.

regards
Wolfgang

Tags (2)
11 Replies

Re: max performance / throughput of site2site-VPN

0 Kudos
Wolfgang
Silver

Re: max performance / throughput of site2site-VPN

Thanks Günther,

we checked these sk twice but with no success.

Wolfgang

0 Kudos

Re: max performance / throughput of site2site-VPN

when using 10Gb interfaces this should be achievable, if however you are using port channels / bonding, that will be your limiting factor. On the 1Gb interfaces even if you combine 5 of them in a port channel/bond, there is still a hashing mechanism that will assign the traffic to a specific port in the channel, this is where you run into the single session limit of the port channel of 1 interface.

Regards, Maarten
Wolfgang
Silver

Re: max performance / throughput of site2site-VPN

You’re right Maarten. A single connection will be run everytime over only one interface and is processed only on one core.

But we are using 10G interfaces. It looks like this is a limitation oft he system, maybee not the hardware but the software.

Wolfgang

Admin
Admin

Re: max performance / throughput of site2site-VPN

If it's between a single source to a single destination (independent of VPN), then this is expected behavior.

We call this an "elephant flow."

Re: max performance / throughput of site2site-VPN

Is this a limitation per tunnel?

So if we are connecting to sites does it make a difference, if we choose one tunnel per GW/network/host?

0 Kudos

Re: max performance / throughput of site2site-VPN

It is not a limitation per VPN tunnel specifically, so modifying the VPN Tunnel Sharing value will have no effect on your issue, the limitation is per individual connection (and a single VPN tunnel can carry multiple connections of course).  All the packets of a single connection (TCP or UDP) can only be handled by one core (whether it is a SND/IRQ or Firewall Worker), so in the case of an elephant flow all those packets will hit a single core and cannot be spread around to different cores to relieve the load.   Trying to spread the packets for a single elephant flow connection around to multiple cores raises the specter of out-of-order delivery, which if it happens will cause TCP to sharply curtail its send rate due to the congestion/problems it is seeing in the network flow and is an unmitigated disaster from a performance perspective.

Should this single-core limitation per connection ever be lifted, there would need to be some kind of reordering mechanism on the firewall to ensure packets are delivered in order.  However doing so would bottleneck the flow of a connection's packets down to the speed of the slowest or most congested core handling packets for that connection, as packets that have already been inspected and are ready for transmission get held/queued waiting for earlier packets to clear inspection on the slowest or most congested core.  Ugh.

 

"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com

Re: max performance / throughput of site2site-VPN

OK, thanks, that answers my question!

0 Kudos

Re: max performance / throughput of site2site-VPN

Hello Wolfgang,

6,5GBbs is total throughput that an appliance can process via VPN. This applies to all summated connections. 

900Mb/s for a "elephant flow". It's a very good value. 

I think the VPN running over an internet interface. This makes runtime problems of the connection critical in the WAN. If you send a TCP packet over the VPN route, there are still runtimes in the WAN to consider. This is related to the windows sizing for TCP connections. This means that you send some packets in one direction and have to wait for the answer packet. It's always slowing you down on the WAN line and it's not a Ckeck Point problem, it's a WAN problem.

If you want to speed up a single connection "elephant flow", you need systems that optimize TCP and optimize protocols. I don't want to advertise for other manufacturers in the Check Point forum, but have a look at Riverbed Steeheadˋs. 

Intel‘s AES New Instructions AES-NI is a encryption instruction set that improves on the Advanced Encryption Standard (AES) algorithm and accelerates the encryption of data in many processor familys. Comprised of seven new instructions, AES-NI gives your environment faster, more affordable data protection and greater security.

 

For more informations about AES-NI see this article:

R80.x Performance Tuning Tip - AES-NI 

You may have other problems on the WAN interface that you can't control:
- fragmentation > check MTU size
- lost VPN packets > and therefore lost TCP packets and therfore TCP retransmissions

- packed in wrong sequence  > and therfore TCP retransmissions

Regards

Heiko

Wolfgang
Silver

Re: max performance / throughput of site2site-VPN

Dear Heiko,

thanks for your response.

Our setup has no internet connection, we used leased lines between our gateways. No MTU issues, no fragmentation problems, no CPU 100% utilization and best suggested parameters for the IPSEC-values.

At CPX I had some interesting discussions with CheckPoints R&D guys.

Now I understand, 900Mb/s for one connection is a very good value. But with CheckPoint there is no chance to get more for this type of traffic, this is a really limitation.

With more connections and more then one VPN-tunnel we can get much more throughput. I found out in the last days, CheckPoint is one of the most valuable vendor for this kind of traffic but for the special case with only one connection we have to lock for another. We found one, another then you mentioned Heiko.

Another thing I learned..... CheckPoint is not alone with the problem of a single connection and „elephant flows“, but other vendors are not really better and most of them are much slower.

thanks for all suggestion,

Wolfgang

Re: max performance / throughput of site2site-VPN

Which alternative vendor did you finally use?
0 Kudos