Performance of a single connection ?

[Wolfgang]

Dear CheckMates,

I had a question regarding maximum performance for a single connection.... 

We had some servers they generate a lot for traffic. host A <=> host B approximately 2Gb/s.

Without any deep checks this connection is going via SecureXL Path and the CPU for this connection is going to 100%.

All other connection they are staying on this CPU are effected and slowed down.

Multiqueue does what it should and starts new conections on other CPUs, but no existing one is moved to another CPU.

Same connection with enabled IPS brings 100%CPU after 350Mb/s and connection goes the PXL path. We played with "disable IPS under load". IPS will be disabled but too only for newly created connection. The existing connections are not effected.

My questions....

Is there a possibility to accelerate one connection, in best case with enabled deep inspection blades ?

Or maybee is there a way to move connections from one CPU to another if this CPU is at 100%, leaving the high traffic connection alone on the high CPU ?

Are there any known values for maximum bandwidth for one connection on an appliance ?

(it looks like 350Mb/s is a real world value for one connection with enabled deep inspection blades like IPS, APPCL ect.)

hope someone can help

Wolfgang

[Daniel_Taney]

Did you try creating an IPS exception for that specific communication? In R80.10, it would be under Threat Prevention -> Exceptions. I'd make the rule as specific as possible for source, destination, and port. For Protection/Site/File/Blade, leave Any selected. Set the Action to Inactive. I believe this should help resolve that!

[Timothy_Hall]

An IPS exception will save CPU overhead in the PXL path and probably provide a boost in speed, but will not allow the traffic to take the SXL path for full acceleration.

To make the subject traffic eligible to take the SXL path, define a Threat Prevention (TP) rule at the top of your TP policy matching the traffic, then specify a TP profile in that rule that has IPS UNchecked (I call this a "null TP profile" and it is covered in my book).  Assuming that the connection is not matching a APCL/URLF rule (the subject traffic cannot match any rule invoking APCL/URLF inspection or it will always go PXL), the traffic should then go SXL unless some other violation factor such as fragmentation mentioned by Daniel Taney‌ is present. 

Don't worry about SecureXL templates getting disabled at a certain rule (i.e. "Disabled by rule #" appears in output of fwaccel stat) as that only improves Firewall/Network Policy Layer rule lookup performance, but does not impact what path (SXL/PXL/F2F) the traffic ultimately will take.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Daniel_Taney]

Thanks for clarifying a number of my points! I think I missed the "Null TP Profile" suggestion in your book. I'll have to go back and revisit that section. 

[Daniel_Taney]

Do you still need the "Null TP" rule if IPS is the only blade enabled on the GW?

[Timothy_Hall]

If you want the traffic to potentially go SXL, yes you'll need a null profile, unless you are using the built-in Optimized/Default_Protection profile which should still allow SXL handling as long as no signature with a Performance Impact higher than Medium has been manually enabled or otherwise activated in that profile.  Be sure to check the Inspection Settings applied to the gateway as well for enabled settings with a High or Critical Performance Impact rating.

Or if you have no TP rule matching the traffic at all that will potentially allow SXL handling as well.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Wolfgang]

Dear Daniel,

the problem with SXL and exception is described in sk110519. That's one of the limitations....

As Timothy described, exception is not the solution. If you want to go sure that the SXL path is used you need a rule with a profile without any activated feature. Like this one:

Wolfgang

[Wolfgang]

Dear Tim,

thanks for your explanations.

Following my original post.... Dou you know a limitation or are there any experience about the possible throughput of a single connection. What will be possible if it goes SXL path and what if it goes the slowest path ?

regards

Wolfgang

[Daniel_Taney]

Here's an example of a rule I have to ensure Threat Prevention doesn't step on iSCSI Replication across Firewalls:

[Wolfgang]

Hi Daniel,

that's what we did. We had some NetApp storage systems they do the replication between more then one system. But they are very powerful. Without IPS we got 2Gb/s throughput for this connection and a 100%CPU. The storage systems can provide more throughput, but the limitation factor is the firewall at the moment.

regards

Wolfgang

[Daniel_Taney]

Ok... so you already have a Threat Prevention Exception rule in place? Sorry if I missed that in your original post.

A couple of other quick thoughts... do you have a rule somewhere in your rule base above the iSCSI one that is blocking SecureXL? It is getting harder for that to happen since R80.10, but it never hurts to check. Run fwaccel stat and make sure Accept Templates doesn't indicate "Disabled from Rule x". If it does indicate SecureXL is being blocked by a rule, make sure your iSCSI rule isn't below it. If it is, either move it above the rule that is stopping SecureXL or try to address why that rule is problematic for SecureXL processing. 

If you don't see SecureXL being disabled by a rule, I'd run the following to see what path your iSCSI traffic is processing in: fwaccel conns | grep <src ip> | grep <dst ip> 

I believe all dots in the Flags column indicates it is processed in SecureXL, an F means the Firewall (slowest) Path and both an F and S mean the medium path. 

There is a debug method to see why a certain connection is taking a certain path, but that could potentially cause performance issues on the Cluster while it is running. If it gets to that point, I'd recommend engaging TAC or at least running the test off-hours. 

Another reason for high CPU could be packet fragmentation. Check fw ctl pstat and fwaccel stats -p for packet fragments. If you see a high number, run the command a couple times and see if it is consistently incrementing. If packet fragments are a problem, I'd run a packet capture on the Gateway and determine the cause of the fragmenting packets. If your iSCSI traffic is using Jumbo Frames, those frames could be getting fragmented and split up on the Firewall. 

Hope this helps!

-Dan

[Daniel_Taney]

Sorry for the double replies.. Can you also let us know what type of Check Point appliances are involved and what versions they are running? What types of Interfaces are you using (onboard 1GB, add-on 1GB copper, 10GB, 40GB, etc...)? Are you using bonding groups for link aggregation (802.3ad)? Which interfaces have Multi-Queue enabled?

[Wolfgang]

Deal Daniel,

thanks vor your writing.

We don‘t have a Problem with SecureXL. The overall performance of the gateway is fine.

The gateway we are talking about is a 15400 appliance. We find the throughput values as i wrote in my original post.

playing around with some other hardware  ( 13800, 15600, 21000 ) shows other values but not significant better.

I think there is a limitation for a single connection. As I understand how CheckPoint works.... all traffic for this single connection will be handled only by one CPU. If this CPU goes to 100%, all other connections handled by this CPU, will be affected.

I think the performance of the CPU is the limiting factor? That‘s what I want to know. And maybee a way to not affect other connections if one is consumption so much load. At the moment we are speed down these with QoS, but this is only a workaround.

regards

Wolfgang

[Maarten_Sjouw]

We also have a coupl of customers that run these type of backup synchronization programs, when this stuff is run in a DC you could try to avoid getting the traffic through the FW altogether by using a backup VLAN (maybe private VLAN's is security is the issue).

Do you have Dynamic dispatching turned on? this will at least make sure that new connections will no longer be handled by that one CPU. Some of these Backup systems also seem to use CIFS for the file transfers, which means you would also need to work on specific CIFS channels to exclude from inspection.