VPN performance on a 3800

[dj0Nz]

Dear community,

I am currently investigating an issue on a CPSG 3800 cluster running only S2S vpns. Throughput is limited to roundabout 300 Mbps because CPU 0 is contantly at 100% load. Besides normal SND tasks, there is the process show below which is causing the load:

Does any of you had similar issues and a solution?

Cheers,
Michael

[G_W_Albrecht]

Question was: why did you use sha256 in P2 ? sk174848 has MD5...

[dj0Nz]

Screenshot from sk174848...

[the_rock]

Seems to me you did everything exactly right.

[dj0Nz]

Update: Traffic limit now at ~800 Mbps with SHA1 hashing. Customer still not satisfied (me neither). Still only one core at 100%, others are ~90% idle. 

[the_rock]

Hey @dj0Nz 

Just wondering, that screenshot you sent, is that AFTER TAC asked you to follow sk174848?

Andy

[dj0Nz]

Hi Andy,

we didn't ask TAC as we configured the VPN but we were following the sk you mentioned...

[the_rock]

Keep us posted, super interested how this gets solved.

Andy

[Chris_Atkinson]

Can you please clarify how many SND are currently set/allocated?

[dj0Nz]

Currently, there are 3 SND cores and 5 workers with dynamic balancing activated.

[Lesley]

I assume that CPU 0 is your SND? How many you have? The other cores are FWK's? 

For me it looks good , the performance. You pull the traffic via one CPU only, as I can see. I think the datasheet speed is for when more then 1 CPU is used. I have never seen 1 CPU carry all the speed that is specified in the datasheet. 

[dj0Nz]

Yes, CPU 0 is one of three SND cores.

But for the customer, this doesn't look good because the machine is specified with 2.75 Gbps of IPSEC traffic and he doesn't care how many CPUs are involved. 

As a technician, I know that the CPUs used in this box are low end Atoms and I would not expect more than the ~800 Mbps we reached today but customer is right here.

Who would accept a glass of water if he ordered a pint of ale?

[Lesley]

I think this is not a technical issue anymore, more customer expectation.

Now you pull the traffic via one SND and get the performance you should get.  

[Timothy_Hall]

Only other thing I'll add is that the new Hyperflow/pipelining feature in R81.20 requires 8+ cores be present on a Check Point appliance.  All current Check Point appliances with 8 or more cores support Hyperflow, *except* the 3800 which sk178070  states is explicitly not supported even though it has the minimum 8 cores.  Hmm...

[Lesley]

Hyperflow is for the deeper inspection (traffic on the FWK). This traffic goes via the SND. 

[Timothy_Hall]

Correct, but the point is that it has 8 cores yet can't support Hyperflow?  What is different about those 8 cores on an 3800 vs another appliance?  

[Lesley]

This model uses older CPU's.

For topic starter here is more background information.

Topic has been discussed here: https://community.checkpoint.com/t5/Security-Gateways/Check-Point-CPAP-SG3800-and-expected-performan...

And here:

https://community.checkpoint.com/t5/Security-Gateways/max-performance-throughput-of-site2site-VPN/m-...

[Bob_Zimmerman]

The 15600 uses two Xeon E5-2630 v3 processors, which were introduced in 2014q3 and which has since been discontinued. It supports HyperFlow. The 3800 uses an Atom C3758 processor, which was introduced in 2017q3 and which is still being manufactured. It does not support HyperFlow. This isn't about the age of the processor.

My bet is it's down to the TDP. At 25W with no turbo boost, the C3758 is one of the lowest-power eight-core amd64 processors available. Even eight-core laptop chips generally have double the TDP. Per-core TDP strongly correlates to per-core performance. It probably just isn't fast enough for HyperFlow to provide a good experience.

[AmitShmuel]

As stated by @Lesley, this model uses older Atom CPU, which lacks certain optimizations. Supporting it is currently under evaluation.

[dj0Nz · Accepted solution]

Okay problem is finally "solved": Check Point provided a POC hardware (6400) with better CPUs but lower VPN throughput according to datasheet. But single thread performance matters in this case. Bandwidth is now at interface maximum (1 Gbps) with CPU 0 load at about 30%.

Conclusion: Either "someone" fixes the datasheets or performance figures must be divided by ten...

[PhoneBoy]

I don't believe our datasheets include VPN performance numbers at present.
For specific guidance on that, you will need to reach out to your Check Point SE.

Having said that, R81.20 should have significantly better VPN performance:

• Scalable VPN performance - 3 times faster to process simultaneous Remote Access and Site to Site VPN connections.
• Major performance and stability improvement for Remote Access VPN and Site to Site VPN that delivers a significantly greater capacity for VPN tunnels.
• Extended Security Gateway certificate validation capabilities for quicker authentication.
• Resilient VPN architecture - multi-process architecture to handle IKE negotiations in dedicated scalable daemons, providing unprecedented resiliency.

[sgvadtbdrfyhng]

---

@PhoneBoy wrote:

I don't believe our datasheets include VPN performance numbers at present.[...]

---

Are you certain? I was under the impression that on page 3 of each appliances' Data Sheets under Specifications / Performance / RFC 3511, 2544, 2647, 1242 Performance (Lab), there is clearly a line specifying "VPN AES-128 (Gbps)" followed by a numeric figure. If that is not VPN perfromance numbers, then I do now know what is.

Obviously (for an experienced firewall admin) you won't get those figures unless under VERY specific conditions, but barely getting 10% of that figure IS a problem.

I also agree that NULL encryption might throw a wrench in the VPN stack's mechanism, but it does appear as if the 3xxx series may have some serious throughput issues, due to some form of software failure.

[the_rock]

Here is something that most people may not even realize...numbers you see on data sheet for any appliance for any vendor really (not only CP) are NOT what is even close to reality. Those numbers are usually only true if you dont enable any additional blades and say for VPN, since you need to have vpn blade on, you leave everything else by default.

It does not even take into account creating super basic security rules.

That is 100% the truth.

Best,

Andy

[Timothy_Hall]

Agree with The Rock here, the published VPN number also will not be anywhere close to achievable with a single VPN tunnel.  VPN handling is to some degree multi-core (sk118097) and can be spread around, but everything going through a single VPN tunnel is more or less limited to the speed of one processor for inspection and another processor for encryption/decryption.  Hyperflow/pipelining (which allows potentially many cores to handle/inspect the packets of a single connection in R81.20) does not apply to VPN traffic operations.

[the_rock]

@sgvadtbdrfyhng 

Not to sound ironic now, but I will tell you a quick story, maybe not the best comparison, but you will get an idea. When I bought my Mazda 6 back in the day, guy was showing me all the bells and whistles on it and he goes "As you can see, it has an amazing fuel efficiency, 100kms highway on 6.1 litres" and I said to him "Come on, we all know thats nonsense, no offence" and his face turned a different color lol

He knew I was 100% right, but its a "script" they have to go by. Then, few years later, I was in Japan, met with a guy who lived there (we used to work for same company), told him the same story, he took me to one Mazda dealership in Tokyo and we saw EXACT same Mazda 6 car, year and model and he translated for me what it says in fuel efficiency and bam, it showed 100 km highway driving and you spend 8.45 litres.

But then again, thats Japanese culture, they are ALWAYS 100% honest and would never exaggerate or inflate anything, no matter how small or big it might be.

Anyway, food for thought as they say..

Best,

Andy

[PhoneBoy]

Those numbers occur only under ideal testing conditions and with the specific encryption settings specified.
Some settings benefit from hardware acceleration in the CPU, others do not.
This can make a significant difference in observed performance.