90%+ PXL traffic despite my efforts

[Justin_Sanders]

Hoping someone might be able to provide some insight on a performance issue occurring on multiple clusters in our environment. 

Gateways are running R77.30 with R80.10 SMS. Latest Jumbos applied to all. Most of them are running firewall and IPS blades only. IPS profile is set to optimized. No exclusions. I've gone through and made sure that no High or Critical performance signatures are active on the profile. Gateway is currently set to Detect only in the IPS section of the cluster object.

Vast majority of traffic is HTTP, HTTPS, DNS, and RTP. cpview shows HTTPS and UDP as the top traffic types hitting PXL under Advanced-Network-Path.  Less than 10% of traffic is hitting F2F. 

 If I disable IPS blade as a test, SXL accelerates 80%+ of traffic and CPU drops in half, so IPS is clearly impacting performance. Disabling IPS is not an option, and neither is significantly reducing the protected scope.   

I've read through Timothy Hall's max power a couple of times now and nothing in there is jumping out as something I missed in terms of basic IPS tuning. 

A few questions really stick out in my mind 

1. If IPS is set to "Protect internal hosts only" and at least 33% of the traffic is destined to an external interface with correct topology definition, how is it that over 90% of traffic is hitting PXL? It would seem that this couldn't solely be a result of IPS. 

2. Does setting the gateway into detect mode in the cluster object cause any sort of issue (rather than configuring detect on the signatures and letting policy control it) ? And yes, I understand that prevent is desired, but I still need to do proper analysis on the traffic to ensure production traffic is not impacted. 

3. I have yet to run a debug to look for why specific traffic is hitting PXL due to difficulty in getting a maintenance window. Is there even any benefit to this? IPS is the likely culprit and I don't need a debug to figure that out; but will it provide anything other than sent to PXL for IPS?

4. The Max Power book makes it seem that all medium or lower performance impact signatures on IPS are eligible for acceleration. If all High and Critical signatures are inactive, how can I explain this behaviour? Am I wasting time tuning the Optimized profile when gateways are still R77.30?

5. Is there any usual culprits in the firewall blade that I should be looking at? Connection templates are disabled early on in the rule base but my understanding is that it will still accelerate traffic after the initial policy decision. 

Any thoughts or guidance is appreciated. 

[Timothy_Hall]

Unfortunately as mentioned in my book the extremely limited R77.XX "IPS Protection Scope" setting is going to hamstring your efforts here.  The high CPU load is not really caused just by traffic being handled in the Medium Path, but the fact that traffic between high-speed LAN interfaces (internal to internal and internal to/from DMZs) is getting sucked into the Medium Path.  Even with "Protect Internal Hosts Only" set, the entire connection with its c2s and s2c streams must be inspected in the Medium Path.  It is not possible to have one flow of a single connection heading to External inspected in SXL while the flow heading Internal is handled in PXL.  IPS Exceptions don't really help here either as they only change the decision rendered by IPS (Prevent vs. Detect) and not which path the connection is handled in.

IPS Protections with a performance impact of "Medium" are about 90% handled in the Medium Path.  So I guess you could try to disable those.

Detect will cause more CPU load than Prevent, but does not influence path selection by the firewall.

With R80.10+ gateway you can define extremely granular rules specifying where you want a particular IPS Profile applied, and can invoke highly-optimized IPS Profiles for high-speed LAN traffic but much more strict IPS Profiles for lower-speed Internet traffic.  You can even do the "null profile" trick mentioned in my book to exclude certain traffic from IPS inspection completely and make it eligible for full acceleration in the SXL path.

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[Justin_Sanders]

Hello Timothy 

First off, I just want to say thanks for taking the time to respond to questions on here and 5 stars for your book. 

I'm a little confused now based on your response. From the section of your book entitled "Signature Performance Impact Rankings" it says " ... the equivalent default on an R80+ SMS is called Optimized. All signatures enabled in these profiles can be handled 100% in the accelerated path" 

While I certainly understand the benefit to applying different profiles to their relevant protection scopes, this statement (and the majority of that chapter) leads me to believe that using the optimized profile and ensuring no high\critical performance signatures are enabled at a global level should allow large amounts of IPS traffic into accelerated path. From there we would further tune the protection scopes to signature sets relevant to that zone, in order to reduce CPU time spent in Medium path. If this were the case, then I would expect far less than 90% of traffic to hit PXL on a gateway running Optimized IPS and firewall blade only. Am I over simplifying this? 

Maybe I am misunderstanding how IPS works. As an example, it I have a theoretical profile with a protection scope of "any" that has 5 low impact signatures related to HTTP and a medium impact signature related to DNS. Does all traffic get sucked into medium path, or just the traffic that would match the DNS protocol?

As for "Protect Internal Hosts Only vs Protect All" , it makes absolute sense that traffic would need to be inspected in both directions, but does this setting not determine the activation of a signature depending on egress interface of the initiated connection? IE. Traffic to internet via external would not invoke IPS (in either direction), whereas traffic inbound from internet would invoke IPS (in both directions)

Thanks again 

[Timothy_Hall]

The "Optimized" profile is close to Default_Protection but not completely equivalent as the book states.  Notice that on p. 300 this appears:

IPS Protections with a “Medium” Performance Impact are processed at least 90% in the Medium Path

which is definitely correct.  On most firewalls there are other blades enabled that force Medium Path inspection for the vast majority of traffic, so at that point all you can do is try to save CPU overhead in the Medium Path as much as possible.  In your case there is just IPS.  In your example about a DNS signature, yes port 53 traffic will be PXL while all other traffic is potentially eligible for SXL path.  Please read my response in the thread below about how SecureXL calculates its ranges in R80.10 and earlier in an attempt to initially figure out what it can handle in the SXL path vs PXL/F2F:

https://community.checkpoint.com/message/26849-re-high-cpu-after-upgrade-from-7730-to-8010?commentID... 

But once again you are focusing on getting lots of traffic to go through the SXL path which is pretty difficult; all you can really do is make as much traffic as possible "eligible" for acceleration without sacrificing too much deep inspection, and at that point try to save as much CPU as possible in the Medium Path. 

The big CPU impact you are seeing is mainly due to the lack of granularity with the "IPS Protection Scope" setting.  The initial direction of connection initiation does not matter, see this excerpt from my book:

Protect Internal Hosts Only – When selected for an R77.30 gateway, IPS protections are only applied to packets traveling to a network defined as “Internal”.

“Internal” in this context means any interface defined as Internal in the firewall/cluster object’s topology anti-spoofing screen (and includes DMZs as well). This also includes traffic passing from one Internal network (like a DMZ) to any other Internal network. It does not matter whether the connection was originally initiated from an External or Internal interface; any packet attempting to leave the firewall towards an Internal interface will have IPS protections applied. Traffic whose destination is an External interface (typically going to the Internet or perhaps a site-to-site VPN utilizing the Internet) will not have IPS protections applied with Protect Internal Hosts Only set.

So all this setting does is save CPU in the Medium Path, it has no bearing on which path the traffic will take.  The IPS Analyzer tool Phoneboy mentioned can help identify which IPS signature could be causing large amounts of CPU overhead.

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[PhoneBoy]

This might be a situation where the IPS Analyzer tool will help.

IPS Analyzer Tool - How to analyze IPS performance efficiently‌