SecureXL - No accelerated connections

[CP-NDA]

Hi,

We are facing an issue with SecureXL

First of all the output of fwaccel stat were indicating that Accept Templates were disabled by Firewall. After investigation we found that QoS module were active in the Package but blade were disabled on the Gateway. Seems that disabling QoS from the package improved the situation as here is the output now

# fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+---------------------------------------------------------------------------------+
|0 |SND |enabled |eth1,eth1-01,eth1-02, |Acceleration,Cryptography |
| | | |Sync,eth1-03 | |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,3DES,DES,AES-128,AES-256,|
| | | | |ESP,LinkSelection,DynamicVPN, |
| | | | |NatTraversal,AES-XCBC,SHA256, |
| | | | |SHA384,SHA512 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled

# fwaccel stats -s
Accelerated conns/Total conns : 66/29137 (0%)
Accelerated pkts/Total pkts : 68884176/69598302 (98%)
F2Fed pkts/Total pkts : 714126/69598302 (1%)
F2V pkts/Total pkts : 476266/69598302 (0%)
CPASXL pkts/Total pkts : 817/69598302 (0%)
PSLXL pkts/Total pkts : 68672216/69598302 (98%)
CPAS pipeline pkts/Total pkts : 0/69598302 (0%)
PSL pipeline pkts/Total pkts : 0/69598302 (0%)
CPAS inline pkts/Total pkts : 0/69598302 (0%)
PSL inline pkts/Total pkts : 0/69598302 (0%)
QOS inbound pkts/Total pkts : 0/69598302 (0%)
QOS outbound pkts/Total pkts : 0/69598302 (0%)
Corrected pkts/Total pkts : 0/69598302 (0%)

As you can see accelerated packets is fine but accelerated conn remain to 0%

What could be the reason of this poor acceleration rate ?

Already involved TAC & followed this SK https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Thank you for your help !

[_Val_]

Actually, your thoughput acceleration rate is very good, 98% packets are accelerated. It is just your template acceleration rate is low. May be nature of the traffic, or something else that breaks templating at an early stage. Which version is in use?

[CP-NDA]

Hi @_Val_ 

Thank you

Indeed Acceleration Rate is very good but Connection rate were flagged by TAC as we are facing big instability & memory leaks issue

We are running on R81.10 T9

[_Val_]

Can you please share your SR via a personal message? Thanks

[the_rock]

I actually installed jumbo take 14 in my R81.10 lab and I can see that sxl stats are much better. Funny enough, did not see any sxl improvements listed though...

[CP-NDA]

Interesting hoewever I really think we need to identify the root cause with TAC without upgrading as it's getting a "traditional answer"

[_Val_]

Not really. JHFs contain lots of fixes, memory leaks included. If you are not on the latest GA Jumbo, it is a very good idea to install it.

[CP-NDA]

Indeed but we have multiple case opened since R80.40. TAC always request for upgrade but at the end of the day same issue occured without finding the root cause. So it's very important to understand the cause and fix it definitvely.

We are not trusting upgrade as the single solution for these issues

[_Val_]

I respectfully disagree with approach.

Having lates Jumbo is actually helping with eliminating all known and fixed bugs, leaving only new and unknown to be investigated. This literally narrows the investigation field, makes TAC processes more effective, and saves times both to you and your support counterpart.

I would understand if you want to stay behind with more tested and proven good versions, but you are already running the latest and (yet) non generally recommended release. Running it as vanilla version does not make sense.  

[the_rock]

Well, we can all agree to disagree...thats what makes this world go round, right? ; ). I see the point you make @_Val_ , but I also logically see the point @CP-NDA makes. I can only speak from my own experience with this and can certainly say that it was not always a winning situation having latest jumbo installed.

[CP-NDA]

It's also very difficult to always find windows maintenance as these upgrade are most of the time disruptive for production environement

After 3 tries and when issues are still occuring we are looking for other suggestion than just an upgrade

I'm also generally ok with proactive upgrade but in this specific case we really need to find other suggestion for the customer even more when previous upgrades were recommended by TAC to solve these issue without success

We usually only installed GA version

[_Val_]

Yes, @CP-NDA you are right, T9 is GA jumbo, my mistake.

Disregard my previous comments, I misread your post and thought you are running the initial release. 

[_Val_]

Well, @the_rock what can I say... Each jumbo has thousands of fixes. It is really unnecessary to reinvent the wheel every time you stumble upon a bug on an outdated version, which is already eliminated.

If you want to get your production systems up and running as fast as possible, I would suggest following support best practices. And if you want to help Check Point debugging the code, you better join our QA team 🙂

The rest is up to you. Remind me your statement above the next time you complain about TAC efficiency, pal 🙂

[CP-NDA]

Understood

Please not that for the moment TAC doesn't suggest the upgrade at the moment and they are still investigating hoewever I've the feeling that they will do this suggestion... For a new issue I'm quite ok with this but with persistent issue I'm expecting more interactions with support.

[the_rock]

Well, I cant say much more either brother :-). Im not complaining at all, just making a statement, my own opinion. I would not go as far to say each jumbo has thousands of fixes (thats really a huge overstatement), but I can tell you that many times in the past, I had cases where customers did have latest jumbo installed and support approach was exactly the same, really no difference. On the other hand, few TAC people even told me before, which I agree with wholeheartedly, that it wont make much difference installing latest jumbo, unless there is specific fix included. To me, there is lots of logic in that sort of statement.

Anyway, just my 2 cents.

[CP-NDA]

No worries I will keep you updated

Case is escalated and they suggest to clear connections tables

Not easy in a prod environement but we need to progress. Not sure that clearing connection table will improve the situation as we don't see any new connection in the fwaccel stats -s

[the_rock]

Im pretty positive that wont do anything...Im not sure either what clearing conn table would improve here. I mean, you can try, but I would be shocked if it did anything in your case.

[CP-NDA]

I'm also 100% that will not have any impact. We have new connection all the time an number is blocked at 65 since yesterday 😅

I can not reject all TAC recommandation so I will request a downtime to do that and maybe combine this with a new upgrade... It's difficult to explain to a customer when you even don't trust the solution yourself

[the_rock]

Again, agree wholeheartedly with everything you said. The way I look at it is this...if anyone asks you to do something, you are entitled to ask them the reason why, its not much to ask for. If they cant give you a logical reason, that means they dont even believe in what they are asking you to do. 

Just my personal opinion, but I believe thats true for anything in life : - )

[CP-NDA]

Fully agree with you

They asked me that yesterday but for me explanation is not good so that's why we are still in this situation

[the_rock]

Just follow advice from @Timothy_Hall . The man sure knows this topic, among many others.

[genisis__]

Another point to add here, as techies we can debut, upgrade have zooms etc, but lets not forget one thing, any vendor selected is there to deliver a service which affects revenue to a business.  

Revenue and also TCO has to also factor in the amount of time a resource has to keep going back and fourth arrange and justify maintenance windows etc (I know this from my own experiences as I'm sure many others have).

Yes - before calling TAC in general I would say ensure you are running the latest GA Jumbo release because that box is then ticked; get as much info as possible to TAC in one hit.  Then your giving TAC every chance to resolve the issue in a timely manner. 

I would also suggest ensuring you set expectations and back that by how this issue affects the business (would it not be nice to have a SLA with Checkpoint that measures SLA and provides credits back for failure of SLA!).  Then keep on top of them, be firm but polite and don't be afraid to escalate the hell out of the ticket if your not getting the urgency the ticket needs.

[the_rock]

All valid points @genisis__ 

[the_rock]

I agree 100%. I just installed it since its a lab and quite frankly, I dont care if it breaks, I will just build another one 🙂

[Timothy_Hall]

Please provide output of enabled_blades and fw ver.

My guess is that you are using an inline layer in your policy with at least both Firewall and Application Control/URL Filtering enabled in that same layer, then using application objects or simple services with "Protocol Signature" set directly in that policy.  When you do that the templating/connections acceleration rate will instantly drop to zero because SecureXL does not have the ability to handle Protocol Signature enforcement on simple services nor to identify applications, that is handled in PXL.  See my comments here:

https://community.checkpoint.com/t5/General-Topics/TechTalk-Security-Gateway-Performance-Optimizatio...

https://community.checkpoint.com/t5/General-Topics/TechTalk-Security-Gateway-Performance-Optimizatio...

Note that this limitation may not apply in the newest versions, but was definitely there at one point.

[CP-NDA]

Thank you @Timothy_Hall 

I'll request output to the customer but for sure we are running multiple inline layers with FW and APCL enabled

We also have IPS, AV, AB, TE

QoS is not enabled

From my understanding enabling these blades could affect inline layers but not the whole rulebase. We are not using any APCL object in the mother rule of any inline layer so it's strange that it impact so much our rule base.

Also we are running R81.10 T9 so the latest GA version available

[the_rock]

I will tell you as well...again, just my own personal experience, but every customer is different. I had seen cases where is you create say, another ordered layer and enable just specific blade, say URLF, that might make a difference...BUT, the problem could be that CP recommends to always place allow rule at the bottom of layer like that, so not positive it would make huge difference in your scenario, but worth considering. Are you using just one ordered layer with multiple blades or multiple ordered layers? 

[CP-NDA]

We are using one ordered layer with multiple inline layers.

Main ordered layer is just Firewall. Inline layers have sometime APCL enabled.

After that we have 2 Threat Prevention Layer IPS and AV, AB, TE

[Timothy_Hall]

Right the recommendation is to only reference simple services in the top level parent rules, then invoke Medium Path inspection elements such as applications  in sub-rules.  However even doing this will not allow the formation of templates from what I remember.

Try this: Temporarily disable ABOT on your gateway and reinstall policy.  Do templates begin to form?  This limitation  was noted in one of my books which called Anti-bot the "slayer" of SecureXL templates.  If that has no effect try temporarily disabling AV as well.  These features involve IP Reputation checks which cannot be performed by SecureXL and as such accept templates are not allowed, although I thought this limitation went away at some point.

[the_rock]

Good advice. I usually have people to set up parent rules with services any, but might reconsider that.