No Accelerated Packets

[Shawn_Fletcher]

At some point recently we've started to see more high CPU type issues. Usually to fix them i'm failing over between clusterXL members and rebooting (we have a fresh environment we are prepping so i'm just trying to keep these alive for now).

One thing i noticed is they have 0% accelerated packets. I know i did some troubleshooting 1-2 years ago  and definitely did not have this problem as we ended up deploying more SND cores then the default. Recently we've been removing interfaces (migrating them to the new boxes) and also been on an eval license, but i cant think of what else major that would have changed.

fwaccel has been turned on/off  to confirm it doesn't change 

QoS blade has been removed

https inspection policy has been checked for rules with "any" (only in the cleanup and not on the service)

running r80.20 take 103 on 21800 appliances

any suggestions would be appreciated.

[Expert@W5FW1:0]# fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+---------------------------------------------------------------------------------+
|0 |SND |enabled |eth3-01,eth3-02,eth1-01, |
| | | |eth3-03,eth3-04,eth1-02, |
| | | |eth1-03,eth1-04,Mgmt |Acceleration,Cryptography |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,NULL,3DES,DES,AES-128, |
| | | | |AES-256,ESP,LinkSelection, |
| | | | |DynamicVPN,NatTraversal, |
| | | | |AES-XCBC,SHA256 |
+---------------------------------------------------------------------------------+

Accept Templates : disabled by Firewall
Layer SJH-R80-10_21800 Security disables template offloads from rule #1
Throughput acceleration still enabled.
Drop Templates : disabled
NAT Templates : disabled by Firewall
Layer SJH-R80-10_21800 Security disables template offloads from rule #1
Throughput acceleration still enabled.

[Expert@W5FW1:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/0 (0%)
Accelerated pkts/Total pkts : 0/75252241 (0%)
F2Fed pkts/Total pkts : 75252241/75252241 (100%)
F2V pkts/Total pkts : 0/75252241 (0%)
CPASXL pkts/Total pkts : 0/75252241 (0%)
PSLXL pkts/Total pkts : 0/75252241 (0%)
CPAS inline pkts/Total pkts : 0/75252241 (0%)
PSL inline pkts/Total pkts : 0/75252241 (0%)
QOS inbound pkts/Total pkts : 0/75252241 (0%)
QOS outbound pkts/Total pkts : 0/75252241 (0%)
Corrected pkts/Total pkts : 0/75252241 (0%)
[Expert@W5FW1:0]# fwaccel stats
Name Value Name Value
---------------------------- ------------ ---------------------------- ------------

Accelerated Path
--------------------------------------------------------------------------------------
accel packets 0 accel bytes 0
outbound packets 1335432 outbound bytes 1018825432
conns created 0 conns deleted 0
C total conns 0 C TCP conns 0
C non TCP conns 0 nat conns 0
dropped packets 20490 dropped bytes 24839949
fragments received 18326 fragments transmit 82542
fragments dropped 0 fragments expired 0
IP options stripped 0 IP options restored 0
IP options dropped 0 corrs created 0
corrs deleted 0 C corrections 0
corrected packets 0 corrected bytes 0

Accelerated VPN Path
--------------------------------------------------------------------------------------
C crypt conns 0 enc bytes 924086080
dec bytes 179947880 ESP enc pkts 1294161
ESP enc err 53 ESP dec pkts 989381
ESP dec err 236 ESP other err 426
espudp enc pkts 0 espudp enc err 0
espudp dec pkts 0 espudp dec err 0
espudp other err 0

Medium Streaming Path
--------------------------------------------------------------------------------------
CPASXL packets 0 PSLXL packets 0
CPASXL async packets 0 PSLXL async packets 0
CPASXL bytes 0 PSLXL bytes 0
C CPASXL conns 0 C PSLXL conns 0
CPASXL conns created 0 PSLXL conns created 0
PXL FF conns 0 PXL FF packets 0
PXL FF bytes 0 PXL FF acks 0
PXL no conn drops 0

Inline Streaming Path
--------------------------------------------------------------------------------------
PSL Inline packets 0 PSL Inline bytes 0
CPAS Inline packets 0 CPAS Inline bytes 0

Buffer Path
--------------------------------------------------------------------------------------
Buffer path buffers 0 Buffer path bytes 0

TLS PARSER
--------------------------------------------------------------------------------------
RECORD INFO 0

TLS DECRYPT
--------------------------------------------------------------------------------------
TLS INSPECTION 0 TLS HANDSHAKE 0
TLS RECORD LAYER 0 TLS CRYPTO 0

HTTP DISP
--------------------------------------------------------------------------------------
ACTIVATE WS MAIN 0 EXEC NO HTTP CMI CONTEXT 0

WS LITE
--------------------------------------------------------------------------------------
WS TX COMPLETED 0 WS FORWARD TO MAIN 0
WS NOTIFY TIMEOUT 0 WS HANDLE EVENT 0
WS CHUNKED ERROR 0 WS GZIP EVENT 0
WS ADD MAC HEADER 0 WS IS STICKY ACTIVE 0
WS TIER1 JOB ERROR 0 WS TIER1 HAS MATCHES 0
CML MATCHES 0 TOTAL UPLOADED JOBS 0
TOTAL JOBS 0

ADVP
--------------------------------------------------------------------------------------
ADVP FORW TO MAIN 0 ADVP HOLD TIMEOUT 0

QoS Paths
--------------------------------------------------------------------------------------
QoS General Information:
------------------------
Total QoS Conns 0 QoS Classify Conns 0
QoS Classify flow 0 Reclassify QoS policy 0

FireWall QoS Path:
------------------
Enqueued IN packets 0 Enqueued OUT packets 0
Dequeued IN packets 0 Dequeued OUT packets 0
Enqueued IN bytes 0 Enqueued OUT bytes 0
Dequeued IN bytes 0 Dequeued OUT bytes 0

Accelerated QoS Path:
---------------------
Enqueued IN packets 0 Enqueued OUT packets 0
Dequeued IN packets 0 Dequeued OUT packets 0
Enqueued IN bytes 0 Enqueued OUT bytes 0
Dequeued IN bytes 0 Dequeued OUT bytes 0

Firewall Path
--------------------------------------------------------------------------------------
F2F packets 75348494 F2F bytes 13533408305
TCP violations 0 F2V conn match pkts 0
F2V packets 0 F2V bytes 0

GTP
--------------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0

General
--------------------------------------------------------------------------------------
memory used 126883336 C tcp handshake conns 0
C tcp established conns 0 C tcp closed conns 0
C tcp pxl handshake conns 0 C tcp pxl established conns 0
C tcp pxl closed conns 0 DNS DoR stats 0

(*) Statistics marked with C refer to current value, others refer to total value

[Expert@W5FW1:0]# free -m
total used free shared buffers cached
Mem: 64282 46121 18160 0 2071 25279
-/+ buffers/cache: 18771 45511
Swap: 32765 0 32765

[Expert@W5FW1:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 1756688 0 0 0 235586 0 0 0 BMRU
Mgmt:1 1500 0 - no statistics available - BMRU
bond0 1500 0 1151388138 0 0 0 921902175 0 0 0 BMmRU
bond1 1500 0 28277182327 0 0 0 34071723724 0 0 0 BMmRU
bond1.504 1500 0 28229249552 0 0 0 34054857508 0 0 0 BMmRU
bond1.2020 1500 0 11112412 0 0 0 10028082 0 0 0 BMmRU
bond2 1500 0 25660905790 0 0 0 17788824679 0 0 0 BMmRU
bond2.509 1500 0 19197533367 0 0 0 12578461784 0 0 0 BMmRU
bond2.512 1500 0 104759957 0 0 0 92945984 0 0 0 BMmRU
bond2.513 1500 0 89510663 0 0 0 73595762 0 0 0 BMmRU
bond2.516 1500 0 107057766 0 0 0 127599142 0 0 0 BMmRU
bond2.517 1500 0 170148410 0 0 0 142009418 0 0 0 BMmRU
bond2.524 1500 0 5470906146 0 0 0 4313539376 0 0 0 BMmRU
bond2.530 1500 0 1037418 0 0 0 1171393 0 0 0 BMmRU
bond2.532 1500 0 28729 0 0 0 7353 0 0 0 BMmRU
bond2.533 1500 0 15391554 0 0 0 8781283 0 0 0 BMmRU
bond2.536 1500 0 29358 0 0 0 23564 0 0 0 BMmRU
bond2.543 1500 0 29038617 0 0 0 29440308 0 0 0 BMmRU
bond2.544 1500 0 152590939 0 0 0 94244829 0 0 0 BMmRU
bond2.545 1500 0 6446718 0 0 0 5715802 0 0 0 BMmRU
bond2.552 1500 0 45524212 0 0 0 24830566 0 0 0 BMmRU
bond2.2501 1500 0 8154151 0 0 0 9674865 0 0 0 BMmRU
bond2.3000 1500 0 172208452 0 0 0 173699324 0 0 0 BMmRU
eth1-01 1500 0 14311283482 0 0 0 34051959502 0 0 0 BMsRU
eth1-02 1500 0 13965900697 0 0 0 19766219 0 0 0 BMsRU
eth1-03 1500 0 13051129355 0 0 0 13443310944 0 0 0 BMsRU
eth1-04 1500 0 12609777564 0 0 0 4345514160 0 0 0 BMsRU
eth3-01 1500 0 493211045 0 0 0 919678110 0 0 0 BMsRU
eth3-02 1500 0 658177108 0 0 0 2224118 0 0 0 BMsRU
eth3-03 1500 0 1259641420 0 34 34 936281619 0 0 0 BMRU
eth3-03.924 1500 0 9894495 0 0 0 9321621 0 0 0 BMRU
eth3-03.1140 1500 0 1212176281 0 0 0 892163611 0 0 0 BMRU
eth3-03.1183 1500 0 37570646 0 0 0 34797662 0 0 0 BMRU
eth3-04 1500 0 19213483 0 0 0 23779047 0 0 0 BMRU
lo 16436 0 102914177 0 0 0 102914177 0 0 0 LRU

[Expert@W5FW1:0]# enabled_blades
fw vpn cvpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation content_awareness mon vpn

[Expert@W5FW1:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 19 | 1385 | 11154
1 | Yes | 18 | 1304 | 9267
2 | Yes | 17 | 1278 | 8919
3 | Yes | 16 | 1249 | 9203
4 | Yes | 15 | 2159 | 10177
5 | Yes | 14 | 1341 | 9676
6 | Yes | 13 | 1337 | 9265
7 | Yes | 12 | 1645 | 9385
8 | Yes | 11 | 1790 | 10221
9 | Yes | 10 | 1695 | 10121
10 | Yes | 9 | 1692 | 10364
11 | Yes | 8 | 1723 | 11291
12 | Yes | 7 | 1804 | 9945
13 | Yes | 6 | 1237 | 11567

[Expert@W5FW1:0]# fw ctl affinity -l -r
CPU 0: Mgmt
CPU 1: eth3-01
CPU 2: eth3-02
CPU 3: eth3-04
CPU 4:
CPU 5:
CPU 6: fw_13
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 7: fw_12
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 8: fw_11
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 9: fw_10
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 10: fw_9
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 11: fw_8
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 12: fw_7
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 13: fw_6
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 14: fw_5
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 15: fw_4
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 16: fw_3
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 17: fw_2
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 18: fw_1
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
CPU 19: fw_0
in.asessiond pepd wsdnsd rtmd vpnd rad fwd cp_file_convertd lpd pdpd mpdaemon in.geod in.msd in.acapd usrchkd cprid cpd
All: scanengine_b
Interface eth1-01: has multi queue enabled
Interface eth3-03: has multi queue enabled
Interface eth1-02: has multi queue enabled
Interface eth1-03: has multi queue enabled
Interface eth1-04: has multi queue enabled

[Expert@W5FW1:0]# fw ctl multik dynamic_dispatching get_mode
Current mode is On

[Timothy_Hall]

1) Make sure you didn't run these commands on the standby member as 100% F2F is expected there.

2) You must have some kind of DCE/RPC service (or a "mapped" service, or another service that has customized inspection code embedded within it, or possibly some kind of dynamic object) in rule #1 which is disabling accept templating (but is not the cause of 100% F2F).  Move that service down as low as possible in your rulebase.  If you don't have any of these services used, please post a screenshot of rule #1 if possible.

3) As far as 100% F2F it sounds like you have checked the HTTPS Inspection policy for "Any" and have a Bypass cleanup rule at the end, but I would check it again as that is the most likely thing causing the 100% F2F.  The next thing to look at is rules invoking Content Awareness, perhaps even try disabling that blade temporarily and see what happens with F2F as there is a lot of security server process work there that must also have some F2F interaction.

4) You aren't using ISP Redundancy in Load Sharing mode are you?

5) Assuming all that checks out, IPS & Threat Prevention are the next items to look at as there are some IPS signatures that will force 100% F2F.  Try running ips off, wait 5 minutes then check acceleration status again, then run ips on.  To do the same thing with the other four threat prevention blades the commands are fw amw unload and fw amw fetch local.  Note that the commands in this paragraph may expose your organization to attacks, use them at your own risk.

[Shawn_Fletcher]

Thanks Timothy for the suggestions

1) definitely can confirm its not the standby member. I did fail-overs and reboots of both members to confirm that doesn't resolve.

2) we are using dhcp_reply / dhcp_request in the first rules on the gateway. Unfortunately the layout of our policy still needs to get some layers added so this is difficult to move lower for other reasons. Working on an improvement there with our refresh, but this has been an issue for us for a long time.

3) confirmed the only rule with any is the bypassrule at the bottom ANY/ANY service https /https proxy . Disabled content awareness blade - no impact (leaving off for now)

4) No

5) Neither of these worked. I ran IPS off, waited 10 minutes, then added uninstalling the thread prevention for another 10 minutes. Didnt help.

The only thing i can think of at this point that's changed is there is an eval license on these gateways... i wouldn't think this is related but grasping at straws. I think TAC is the next step.

Thanks for your help

[Timothy_Hall]

My next question was going to be if you are using ClusterXL Load Sharing Multicast and you have the Sticky Decision Function (SDF) enabled, but after some reading looks like SDF was removed in R80.20+.  The TAC will be able to run a fwaccel dbg to determine exactly why traffic is going F2F, please follow up with what they find.  You must have some kind of legacy feature or service in use that is causing 100% F2F.

[leobase]

Hi,

we have a very similar issue and we could figure out what triggers the error, but not how to solve it. For this we opened a ticket at TAC.

It seems that something has changed between R77.30 and R80.x with services with limited source ports. You are using "dhcp_reply / dhcp_request" services in rule 1. In this service you can find a limited source port under the advanced tab. And this seems to disable securexl.

We had a self defined service where we limited the source ports and figured out, that this service disabeld the securexl. The same policy on a different gateway with the same self defined service in R77.20 and R77.30 is not affected.

Maybe this helps.

Best regards,

Thomas

[Timothy_Hall]

It seems that something has changed between R77.30 and R80.x with services with limited source ports. You are using "dhcp_reply / dhcp_request" services in rule 1. In this service you can find a limited source port under the advanced tab. And this seems to disable securexl.

The ability for SecureXL to create Accept templates (i.e. "cache" rule base lookups) will be disabled if there is a fixed source port number in a rule's service, but throughput acceleration (what "path" the traffic takes) will not be affected.  This has been the case for as long as I can remember, is still true in R80.40, and was even mentioned in the R80.40 addendum released earlier this week for my Max Power 2020 book . 

I bristle a bit every time someone uses the term "disables SecureXL" without clarifying whether they mean templating or throughput acceleration. 😀

[Shawn_Fletcher]

Just an update

the case is still ongoing with TAC, moving the dhcp rules lower still doesn't allow for templating OR throughput acceleration.

looking deeper into the dhcp_reply /request objects though it was supposed to be supported from r80+ according to an SK.

on investigation it looks like we had created our objects in  and older version and ported them over which made them "custom" instead of the built in ones. we actually had 2 objects for each dhcp-reply and dhcp-reply_   - our custom object had "accept replies" where the built in did not.

switching over to the built in, sure enough accept templates now happy.

Still didn't fix the securexl issue we are having...   TAC still investigating.

[Expert@SJHFW1:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/0 (0%)
Accelerated pkts/Total pkts : 0/102280970767 (0%)
F2Fed pkts/Total pkts : 102280970767/102280970767 (100%)
F2V pkts/Total pkts : 0/102280970767 (0%)
CPASXL pkts/Total pkts : 0/102280970767 (0%)
PSLXL pkts/Total pkts : 0/102280970767 (0%)
CPAS inline pkts/Total pkts : 0/102280970767 (0%)
PSL inline pkts/Total pkts : 0/102280970767 (0%)
QOS inbound pkts/Total pkts : 0/102280970767 (0%)
QOS outbound pkts/Total pkts : 0/102280970767 (0%)
Corrected pkts/Total pkts : 0/102280970767 (0%)
[Expert@SJHFW1:0]# fwaccel stat
+---------------------------------------------------------------------------------+
|Id|Name |Status |Interfaces |Features |
+---------------------------------------------------------------------------------+
|0 |SND |enabled |eth1-01,eth1-02,eth1-03, |
| | | |eth1-04,Mgmt,eth3-01, |
| | | |eth3-02,eth3-03,eth3-04 |Acceleration,Cryptography |
| | | | |Crypto: Tunnel,UDPEncap,MD5, |
| | | | |SHA1,NULL,3DES,DES,AES-128, |
| | | | |AES-256,ESP,LinkSelection, |
| | | | |DynamicVPN,NatTraversal, |
| | | | |AES-XCBC,SHA256 |
+---------------------------------------------------------------------------------+

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled

[Shawn_Fletcher · Accepted solution]

Just an update - after months of troubleshooting, multiple versions, different hardware... the root cause was Wire Mode

We had a community on the mgmt server that had been setup as a test some time ago that had wire mode checked off.

The community object was not used in ANY pushed policies however the fact of having it  on the mgmt server was enough to disable secure XL across 18 gateways managed by that management server.

TAC also provided a kernel setting to disable wire mode

Modify $FWDIR/boot/modules/fwkern.conf using "vi".
Add line "sxl_disable_wire_mode_accel=0" in this file.

hope this helps someone 

[Timothy_Hall]

Yikes, that is an obscure one.  Thanks for the followup, looks like an SK was just created too:

sk170133: Acceleration does not work when using wire mode with SecureXL enabled

This issue could be leveraged as an interesting method to disable SecureXL in R80.20+, either "permanently" or just for the duration of taking packet captures with fw monitor -e.  Set wire mode on a community, install policy, take fw monitor -e capture, disable wire mode, install policy.  I'll need to test this...

[network_inf_op_]

Sorry. What do you mean by " that had wire mode checked off."

I tried to create a new vpn community and wire mode is not checked by default.

GS

[Timothy_Hall]

Wire mode allows VPN traffic that is transiting from one VPN tunnel to another in a route-based VPN scenario to skip inspection on the intermediate firewall; it is assumed that the traffic will get inspected on the destination firewall.  Wire mode is disabled by default and should be left that way unless the above scenario is present.  The OP must have been experimenting with route-based VPNs at some point, the box got checked, and it was left that way.