SecureXL Medium Path Puzzle

[Lincoln_Webber]

I have two data centers each having an open server cluster. The data centers are separated by a WAN. The primary DC's cluster has the organization's Internet link and the following blades enabled: IPS, VPN, APCL/URLF, AB/AV, Identity awareness, Email/antispam. The secondary DC's cluster has the only has IPS enabled (the same IPS profile is applied to both clusters). Both clusters have secureXL and CoreXL enabled (Primary DC with 6 instances, Secondary DC with 3 instances).

We have Veritas netbackup servers at both DCs. Replication traffic between the DCs is accelerated on the Secondary DC cluster (fwaccel conns output shows no flags) but goes to the Medium path on the Primary DC cluster (fwaccel conns output shows 'S' flag).

Disabling IPS on the primary DC (ips off) does not make a difference for this traffic

I have specific source and service definitions for the APCL/URLF & AB/AV rulebases

I have even  disabled the above mentioned blades and it has mode no difference 

The traffic in question is on tcp port 1556

Right now we are using qos on the switches the servers are connected to limit the bandwidth because the replication causes sustained CPU spikes on the cores and affects other services (eg. VPN and Internet browsing)  

I would like to know if anyone has had a similar experience and if any solution was found.

[Timothy_Hall]

You didn't say what version of gateway code you are using but the following should apply regardless.

> (the same IPS profile is applied to both clusters)

IPS is probably not your issue then.

> I have specific source and service definitions for the APCL/URLF & AB/AV rulebases

If you want the backup traffic eligible for the Accelerated path you shouldn't do that in your rulebases.  First off, do you have a "Log Everything" Any Any Any Accept rule at the bottom of your APCL/URLF policy?  You don't need it since the implied cleanup rule is Accept for that policy, so get rid of it.  Next you want the backup traffic to not match *anything* in the APCL/URLF policy at all and "fall off the end" of the rulebase (don't worry it will be accepted by default and not logged).  Using a specific rule to match that backup traffic will just pull it all into the Medium Path.

For AB/AV policy: An exception will not usually make that traffic eligible for the SXL path, it will only change the decision about whether the matching traffic is "bad" and should be dropped or not.  The best way to ensure that the traffic you want is eligible for the SXL path is to put a rule at the top of the TP policy matching the traffic, then in the Action field invoke a profile that has ALL of the Threat Prevention blades unchecked in it.  In R80.10 if you have more than one Threat Prevention policy layer (not common), watch out as a match against more than one TP layer will choose the most restrictive action which will almost certainly not be your disabled profile, unless you have added the disabled profile rule to the top of both TP layers.

> I have even  disabled the above mentioned blades and it has mode no difference 

Once the connection has been assigned a path and it continues to be active, I don't believe disabling various blades and reinstalling policy will suddenly make that existing connection accelerated.  New connections of course will potentially now be eligible for the SXL path, but if you are observing a long-running backup connection and are messing around with turning blades on and off I don't think it will have an effect on acceleration.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Lincoln_Webber]

Hi Tim

The firewalls are running R77.30 with JHF 216

I've actually been reading from your book and have implemented those techniques you listed above (removing the any-any-log rule from the APCL/URLF, creating the exclusion profile with inactive TP blades  and assigning it to the specific netbackup communication at the top of the TP rulebase).

When I said I created specific rules in APCL/URLF, I was referring to rules specific to only traffic I wanted inspected (which does not include the netbackup servers)

Interestingly, there is another protocol (tcp 11000) which is used to occasionally ship large amounts between the netbackup media server and the storage array that is fully accelerated by the same firewall.

[Timothy_Hall]

Some ports (like microsoft-ds/445) are always passed up to the Medium Path because it is assumed by SecureXL that active streaming will be required to inspect it.  1556 is low enough that it may be one of those ports.  Do you have a custom service created for that port?  If so what is it's protocol type on the Advanced screen?  It should None/Blank.

If you've already checked that, it might be interesting to try changing the backup software if possible to use a port higher than 10,000 and see what happens.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com

[Lincoln_Webber]

There is no predefined service for it. There is a custom service in use now and the protocol is 'blank'. In cpview, the firewall seems to resolve it to 'veritas-pbx'. 'irisa' is the

protocol i mentioned previously (tcp 11000). I may have been mistaken earlier when i said it was totally accelerated (I may have seen it on the secondary DC firewall)

focus on the connections between devices having 192.168.x51.238 in the image below

[Timothy_Hall]

So just to confirm, you do not see an Allow log entry at all for this port 1556 traffic from the APCL/URLF blades, right?  Assuming you are logging all your APCL/URLF rules that would indicate that APCL/URLF is not putting that traffic into the Medium Path.  Next steps that will help isolate what is pulling it into PXL:

1) Run ips off, wait 30 seconds then start a NEW backup and see what happens with acceleration status on port 1556. 

2) If still not accelerated, next step is to run fw amw unload and then run fw stat -b AMW to confirm that Threat Prevention is off.  Start a NEW 1556 connection, is it accelerated?  Be sure to reinstall policy to your gateway at the end of this step to turn IPS/TP back on.

3) If still not accelerated you have something else going on, please provide output of enabled_blades and fwaccel stats -p, might also be interesting to run a watch fwaccel stats -p just before starting a new 1556 connection and see what violation counters get incremented when it starts.  You may end up needing to do a sim dbg as described in pages 221-225 of my book to see why SecureXL thinks that traffic needs to go PXL.

4) Last resort may be whitelisting this port in SecureXL as described in this thread:

https://community.checkpoint.com/message/10308-re-enforce-securexl-template?commentID=10308#comment-... 

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com

[Lincoln_Webber]

Thanks Tim,

I'll try to get these done over the next couple of days.

[Ben_Crosthwaite]

Hi Tim,

Thankyou for this. After following your advice I was able to solve my issue of traffic not following the SXL path and getting stuck in the Medium path.

You need to create a specific TP policy with all blades disabled and all in your servers/ports to this policy. 

Simply adding exclusions to the existing TP policy does not move the traffic to the SXL path

Thanks

Ben

[Timothy_Hall]

> You need to create a specific TP policy with all blades disabled and all in your servers/ports to this policy. 

Right, I call this technique a TP "null profile" in my book.

> Simply adding exclusions to the existing TP policy does not move the traffic to the SXL path

Correct, TP exceptions do not impact path selection.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Ben_Crosthwaite]

Hi Tim,

Have purchased your book, probably the single most usefull source of documentation I have read in the last 10 years working with checkpoint especially when it comes to IPS tuning. 

To be honest, I didn't give give enough credit to the differences between Profile-based Disabling vs. Exceptions in your book until I found this tech article to tie it all together.

Thankyou for the response

Regards

Ben

[Timothy_Hall]

I did try to emphasize that major difference between TP "null" profiles and exceptions in my book, and that is part of why Threat Prevention policy/profile tuning ended up in the "SecureXL Throughput Acceleration" chapter.  Access Control policy tuning was able to be placed in its own chapter.  TP/IPS tuning was originally going to be its own standalone chapter, but there were so many references to how TP/IPS tuning changes impacted SecureXL Throughput Acceleration that I eventually relented and just put that material in the same chapter with SecureXL.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Lincoln_Webber]

Tim,

From what i see in tracker, multiple connections are opened every couple of minutes between the devices on that port, so I didn't have to ask the server team to manually restart the operations.

I ran the commands you specified and with 'ips off' and amw unloaded, the accelerated packets shot up to almost 90%.

I then re-enabled IPS and kept amw unloaded, which saw the accelerated traffic settle at about 75%. What puzzled me about this was that uninstalling threat prevention (disabling AV & AB in dashboard) did not have the same effect (the netbackup traffic was stilled being pulled into medium path). Can you explain what the difference is? 

I guess from this we can be quite confident that IPS and APCL/URLF aren't the culprits.

Below are some screenshots of showing info from cpview and fwaccel:

[Timothy_Hall]

After unchecking AV/ABOT and reinstalling policy (including TP), what does fw stat -b AMW look like?

I have a feeling it is related to Anti-Virus, see the "Exclude connections from being inspected by the Anti-Virus policy" section of sk92515: How to configure Threat Prevention Exceptions in R77.30 and lower.  Bottom line is traffic can still get pulled into the Medium Path by AV even if you've explicitly told AV not to inspect it in R77.30.  There are a couple of different exclusion techniques in the SK, see if one of them works for you to enable acceleration without having to completely disable AMW/TP.  The different techniques all look like they are trying to accomplish the same thing, but apparently how they are interpreted by SecureXL varies.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com

[Lincoln_Webber]

Hi Tim,

"After unchecking AV/ABOT and reinstalling policy (including TP), what does fw stat -b AMW look like?"

>>fw stat -b AMW shows  Anti Malware: Disabled (same as running 'fw amw unload')

I had implemented the first option from sk92515 some time ago but traffic was still being pulled into the Medium path

I tried the second option yesterday and there was still no difference.

I had opened a TAC case some time ago to ask for the patch referenced in sk106062 CPU load and traffic latency after activating Anti-Bot and/or Anti-Virus blade on Security ... but the engineer did not provide the patch because he said he didn't see the RAD process over-utilizing the CPU.

I want to reopen this ticket now that we have zoned in on what the possible causes are.

The particular issue in that sk is allegedly addressed in JHF take_43, but i'm not seeing that referenced in sk106062. 

[Timothy_Hall]

OK one more thing to try: Disable all TP blades and IPS on the firewall object, reinstall policy, then reboot the firewall and see what acceleration looks like.   I'm wondering if once the backup traffic is established as needing PXL that a policy install isn't enough to allow it to head back down into SXL without a fresh restart of the firewall, or a complete clobbering via fw amw unload.  Once again I'm suspecting the AV blade, mainly due to the number of caveats I've run into trying to get traffic accelerated when the AV blade is involved.

If that doesn't make a difference or isn't feasible, let us know what you find out from TAC.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com

[Lincoln_Webber]

Hi Tim,

I haven't gotten a chance to execute your recommendations. I have however reopened the case with Check Point who are now asking me to reset the connections table. I have asked for justification because I don't see the relevance.