Performance Improved with SecureXL turned off

[Louise_Coe]

Hi All,

We have been having some performance issues on our Production firewall, so decided to run some Load tests.

Important Info

1. Running R80.10
2. VSX
3. 23800 appliances - clustered in a pair
4. SecureXL enabled
5. CoreXL enabled
6. IPS, IA are blades enabled
7. Running VSLS, VS's are split between the two appliances, 5 active on each, with other 5 in standby
8. Hyper-Threading - disabled
9. Of the 24 cores, 6 - SND, 18 - FWK
10. Mix of 10GB and 1 GB interfaces, grouped in bonds.

Some more specific info on the Load environment we are running the performance tests on.

This has 10 instances on this VS, when running the load test we move all the other VS's to be active on the other Gateway in the pair, so this is the only VS active on this gateway. This is done for "just in case" reason to protect production traffic. This VS has a bond of a pair of 1GB interfaces and no virtual switch.

The issue we get is when we try to push more than 800MB through it, we start to see latency between the load generators and the servers we are sending the load to.

The CPU is around 550% over the 10 instances, with the higher ones being 60 - 70%. We have done quite a lot of work to improve this balance, by sending the information from several IPs via a Load Balancer and now it is fairly evenly spread.

If I turn off SecureXL at this point, the latency disappears and we are able to push the load up to 880MB.

If I then enable drop optimisation and turn SecureXL back on the highest we get is 750MB.

If I leave secureXL on and turn off IPS - not much change, back to 800MB, CPU drops to 450%

I then turn Threat Prevention off, CPU drops to 20% overall, and throughput is 950MB

TP off and IPS on, CPU goes upto 130% and throughput is around 900MB

(Sorry this is so long, we do have a call open with TAC)...

So my question is - do we have an issue with SecureXL, as things improve when it is off and how could I troubleshoot that?

We are also planning an upgrade to R80.20, to get the much needed dynamic dispatcher for VSX...

Thanks....

[PhoneBoy]

Generally, performance should improve with SecureXL enabled, not get worse.

Can you describe in detail what kind of traffic the performance testing rig is generating?
Is the generating traffic TCP/UDP traffic to/from various IPs?

If it's one source/destination IP, then you have yourself an elephant flow and there's not a ton you can do to optimize that situation.

In any case, if you're going to get any help here, we probably need to see output from the Super Seven commands.
https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac...

[Timothy_Hall]

There are three main effects when SecureXL is disabled:

1) All traffic will now be handled in the F2F/Firewall path, and cannot be processed in the more efficient PXL or SXL paths.  Can't see how disabling SecureXL would improve performance in this case, unless a large amount of traffic was fully accelerated in the SXL path and trying to be handled by an overloaded SND.  In the context of VSX, disabling SecureXL will cause all traffic to be processed in F2F by fwk processes assigned to that VS.  So if an SND was overloaded with high amounts of SXL path traffic I guess disabling SecureXL could improve performance by spreading the traffic across multiple fwk processes instead of a single SND (but see below).  Would need to see output of fwaccel stats -s command which is part of the Super Seven.

2) Disabling SecureXL also turns off automatic interface affinity.  So instead of the SoftIRQ NIC processing being automatically spread across six SND cores in this situation, all SoftIRQ processing is slammed onto SND Core #0.  Normally this would result in an overload of Core #0 and massive RX-DRPs across the various interfaces, but considering that all significant processing is now occurring in the fwk instances perhaps the single SND core #0 can keep up, and the usual overhead normally incurred to coordinate operations across six active SND/IRQ cores is now gone.

3) Disabling SecureXL also disables Multi-Queue.  Do you have it enabled?  There is a slight performance overhead incurred when Multi-Queue is enabled to "stick" all the packets of a single connection to the same SND core to avoid out of order delivery.  Can't see how disabling SecureXL (and in this case Multi-Queue with it) would improve performance, but the Super Seven outputs may shed some light on this.