cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post

Traffic not accelerated by Secure XL

Hi,

I have been dealing with the secure XL for a while and cannot have the traffic accelerated as you can see the output below.The problem is the cpus are going over %95 during day time and i think the reason is the secure XL not handling traffic as expected as everything is going through the slow path.

I have been through many topics here and I will put the outputs you may ask.

Just a brief information of the firewall, working with ClusterXL, 8 cpu (2 SND, 6 workers) , OPEN SERVER ( I'm not sure if this could be any issue) , 

This is an external firewall, having DMZ, vpn and internet traffic of users and servers and more as you can think.

#fwaccel stats -s

Accelerated conns/Total conns : 14/79668 (0%)

Accelerated pkts/Total pkts   : 370720/214400236 (0%)

F2Fed pkts/Total pkts   : 211158051/214400236 (98%)

PXL pkts/Total pkts   : 2871465/214400236 (1%)

QXL pkts/Total pkts   : 0/214400236 (0%)

 

# fwaccel conns -s

There are 211889 connections in SecureXL connections table

 

The template number is so low.

# fwaccel templates -s

There are 48 templates in SecureXL templates table

 

# fwaccel stat

Accelerator Status : on

Accept Templates   : disabled by Firewall

                     Layer CL-EXT Security disables template offloads from rule #xxx ( just above the last rule)

                     Throughput acceleration still enabled.

Drop Templates     : enabled

NAT Templates      : disabled by Firewall

                     Layer CL-EXT Security disables template offloads from rule xxx ( just above the last rule)

                     Throughput acceleration still enabled.

NMR Templates      : enabled

NMT Templates      : enabled

 

I downloaded the fwaccel conns table and when investigated we see that most of traffic is about these 4 sources with 1 destination address (exchange related F5 traffic) as nearly 1/3 of the whole table is this connection. 

SourceDestinationDPortPRFlags     C2Si/f S2Ci/f Inst
AX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
CX4436F..A...S......40/3232/40
AX4436F..A...S......40/3232/40
DX4436F..A...S......40/3232/40
CX4436F..A...S......40/3232/40
DX4436F..A...S......40/3232/40
CX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
DX4436F..A...S......40/3232/40
AX4436F..A...S......40/3232/40
CX4436F..A...S......40/3232/40
DX4436F..A...S......40/3232/40
CX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
CX4436F..A...S......40/3232/40
AX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
AX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
AX4436F..A...S......40/3232/40
DX4436F..A...S......40/3232/40
DX4436F..A...S......40/3232/40
AX4436F..A...S......40/3232/40
BX4436F..A...S......40/3232/40
DX4436F..A...S......40/3232/40

 

My question is, how come this traffic isn't accelerated? 

 

Thank you

 

0 Kudos
13 Replies

Re: Traffic not accelerated by Secure XL

How many rules do you have ? 

What is it the version of the MGMT and FW ?

In what rule the traffic stops being accelerated ?

In this rule, what are the services used ?

0 Kudos

Re: Traffic not accelerated by Secure XL

 

Hi,

 

How many rules do you have ? 

327

 

What is it the version of the MGMT and FW ?

mgmt - R80.30 - Build 484

fw - R80.10 - Build 161

 

In what rule the traffic stops being accelerated ?

325

 

In this rule, what are the services used ?

DCE rpc traffic, it is moved in the end not to cause problems for sexure xl.

0 Kudos

Re: Traffic not accelerated by Secure XL

Hi

 

Ok, what blades do you have ? Can you run "enabled_blades" ?

0 Kudos

Re: Traffic not accelerated by Secure XL


fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation mon vpn
0 Kudos

Re: Traffic not accelerated by Secure XL

I believe you have just a few templates because of the many blades that you have. Most of the traffic will pass in more than one blade and it is just accelerated in F2Fed and not in "total connections":

F2Fed pkts/Total pkts : 211158051/214400236 (98%)

0 Kudos

Re: Traffic not accelerated by Secure XL

My guess is you are improperly using the object Any in the Destination or Service of your HTTPS Inspection policy and it is pulling all traffic into F2F for active streaming.  Use object Internet for the Destination (you will also need to make sure your firewall topology is completely and correctly defined to ensure this object is being calculated correctly) and only use explicit services like https in your HTTPS Inspection policy.  You might have an "Any Any Any" cleanup rule at the end of your HTTPS Inspection policy, big no-no.

Another possibility is that all traffic is fragmented due to an incorrect MTU somewhere.  Please provide the output of fw ctl pstat.

Last possibility is that you are using ISP Redundancy in Load Sharing Mode, Cluster Load Sharing with Sticky Decision Function enabled, or are using your firewall as an explicit HTTP/HTTPS web proxy, pretty much everything will go F2F as a result in any of those cases.

If practically all the traffic passing through this firewall is outbound user traffic to the Internet and subject to HTTPS Inspection, the 98% F2F might be legit.

Don't worry about templating rates, totally separate issue that is not the problem.

Could also be something in your TP policy causing the high F2F, we'll deal with that once you check your HTTPS Inspection Policy, fragmentation, and the three features I mentioned.

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
0 Kudos

Re: Traffic not accelerated by Secure XL

https inspection rules are set with the internet object and https service. There isn't any any rule in the end.

I was curious about the any objects as the traffic i mentioned is passing through a firewall rule with any dest and service rule. I will add a specific rule for the traffic.
Could firewall policy with any objects be the problem?

We don't have a load sharing cluster, there is user traffic but not all of the traffic is for user internet access, there might be fragmentation as i have put the output but the first possibility might be the cause.

# fw ctl pstat

System Capacity Summary:
Memory used: 9% (8765 MB out of 96499 MB) - below watermark
Concurrent Connections: 110365 (Unlimited)
Aggressive Aging is enabled, not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 10116661248 bytes in 2469888 (4096 bytes) blocks using 1 pool
Total memory bytes used: 0 unused: 10116661248 (100.00%) peak: 4717843200
Total memory blocks used: 0 unused: 2469888 (100%) peak: 1248346
Allocations: 566230308 alloc, 0 failed alloc, 538487847 free

System kernel memory (smem) statistics:
Total memory bytes used: 14461308200 peak: 15218299528
Total memory bytes wasted: 43889265
Blocking memory bytes used: 58531832 peak: 221029776
Non-Blocking memory bytes used: 14402776368 peak: 14997269752
Allocations: 2125678292 alloc, 0 failed alloc, 2125658331 free, 0 failed free
vmalloc bytes used: 14378068460 expensive: no

Kernel memory (kmem) statistics:
Total memory bytes used: 6862722468 peak: 8889698628
Allocations: 2691710826 alloc, 0 failed alloc
2663951514 free, 0 failed free
External Allocations: 24728832 for packets, 244282717 for SXL

Cookies:
3591315592 total, 2750222193 alloc, 2750204668 free,
2808169679 dup, 2969475638 get, 3163934437 put,
762173033 len, 660318754 cached len, 0 chain alloc,
0 chain free

Connections:
558916069 total, 404068565 TCP, 134111376 UDP, 20105413 ICMP,
630715 other, 8052 anticipated, 0 recovered, 110365 concurrent,
208312 peak concurrent

Fragments:
614867937 fragments, 306839276 packets, 52179 expired, 0 short,
7 large, 2022 duplicates, 1572 failures

NAT:
1829598225/0 forw, -1728821971/0 bckw, 67790129 tcpudp,
32972912 icmp, 433995668-632048567 alloc

Sync:
Version: new
Status: Able to Send/Receive sync packets
Sync packets sent:
total : 893794328, retransmitted : 1740, retrans reqs : 1371, acks : 1780599
Sync packets received:
total : 279528838, were queued : 2908698, dropped by net : 3761
retrans reqs : 748, received 7220298 acks
retrans reqs for illegal seq : 0
dropped updates as a result of sync overload: 0
Callback statistics: handled 7139038 cb, average delay : 1, max delay : 4098
0 Kudos

Re: Traffic not accelerated by Secure XL

Your firewall policy config is unlikely to be the problem causing high F2F, the policy config is related to templating and totally separate.

> Fragments:
> 614867937 fragments, 306839276 packets, 52179 expired, 0 short,
> 7 large, 2022 duplicates, 1572 failures

That looks a bit excessive, try running these commands to see where the fragments are coming from and how many are coming through the firewall live:

tcpdump -eni any '((ip[6:2] > 0) and (not ip[6] = 64))'
or
tcpdump -eni any "ip[6:2] & 0x1fff!=0"

The good news is that fragmented traffic no longer requires F2F in R80.20+, so an upgrade to R80.30 might be in order here.

If you don't see a lot of constant frags with tcpdump it could be Threat Prevention causing the high F2F, to test try this:

fwaccel stats -s (note F2F percentage)

fwaccel stats -r

ips off

fw amw unload

(wait 60 seconds)

fwaccel stats -s (note F2F percentage changes)

ips on

fw amw fetch local

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
0 Kudos

Re: Traffic not accelerated by Secure XL

Hi,

Wont the tcpdump increase the cpu usage with a string you have given? As we are already facing high cpu, i shouldn't try to burst it more,it might cause a problem for us if so.

We were planning to upgrade to R80.30 but we heard some issues and decided to wait for them to resolve.

IPS might be a cause in my opinion as it is used with many protections on.
0 Kudos

Re: Traffic not accelerated by Secure XL

I also checked another cluster and the fwaccel view is slightly different,corexl is handling a portion of traffic but secureXL is not.

# fwaccel stats -s
Accelerated conns/Total conns : 0/11528 (0%)
Accelerated pkts/Total pkts : 0/153816765 (0%)
F2Fed pkts/Total pkts : 99482451/153816765 (64%)
PXL pkts/Total pkts : 54334314/153816765 (35%)
QXL pkts/Total pkts : 0/153816765 (0%)

on the fw i checked the
fw ctl pstat
and the fragmentation is not increasing at all for the live connections.

securexl is on' enabled.
approximately 300 rules and securexl is disabled just before the end of the rule base so similar stuff going on.
version is same with other firewall.

with the enabled blades:
#enabled_blades
fw vpn urlf av appi ips identityServer anti_bot ThreatEmulation mon vpn

it is a 4cpu open server and cpu usage is around %50 for this one.
0 Kudos

Re: Traffic not accelerated by Secure XL

Please run the IPS/Threat Prevention tests in my prior post.  At worst the APCL/URLF blades will drive traffic into the PXL/PSLXL paths, not F2F.  My guess would by IPS for the high F2F but the tests will tell you for sure.

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
0 Kudos

Re: Traffic not accelerated by Secure XL

I will test it during a suitable time.
But what if it is the ips causing the problem, we wont be able to shut it down, we have inactivated to suggested signatures on the recommended guides using high cpu, i don't know how we will manage with all the other signatures.
Do you have any suggestions as if the problem might be the ips?
0 Kudos

Re: Traffic not accelerated by Secure XL

Need the results of the test first before we start speculating on what needs to be adjusted in IPS.

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
0 Kudos