SecureXL is enabled, but the traffic is not accelerated

[Thecoder]

SecureXL is enabled, but the traffic is not accelerated. i restarted securexl.but the result not changed. corexl and ht are active. and we have performance problems.

[Expert@fw1:0]# fwaccel stats -s

Accelerated conns/Total conns : 519/114126 (0%)

Accelerated pkts/Total pkts   : 1306379/17442674 (7%)

F2Fed pkts/Total pkts   : 11676500/17442674 (66%)

PXL pkts/Total pkts   : 4459795/17442674 (25%)

QXL pkts/Total pkts   : 0/17442674 (0%)

[Expert@fw1:0]# fwaccel stats -p

F2F packets:

--------------

Violation             Packets            Violation             Packets       

--------------------  ---------------    --------------------  ---------------

pkt is a fragment                 389    pkt has IP options                225

ICMP miss conn                  80304    TCP-SYN miss conn             5282924

TCP-other miss conn           7127805    UDP miss conn                 1435788

other miss conn                  5652    VPN returned F2F                   15

ICMP conn is F2Fed              24867    TCP conn is F2Fed           167500203

UDP conn is F2Fed               29421    other conn is F2Fed                 0

uni-directional viol                0    possible spoof viol                 0

TCP state viol                 103953    out if not def/accl              5518

bridge, src=dst                     0    routing decision err               87

sanity checks failed                0    temp conn expired                   3

fwd to non-pivot                    0    broadcast/multicast                 0

cluster message                     0    partial conn                    92673

PXL returned F2F               192634    cluster forward                     0

chain forwarding                    0    Tmpl no-match range                 0

Tmpl no-match time                  0    general reason                    306

route change                        0    inbound zone change                 0

outbound zone change                0   

[Timothy_Hall]

Please post the output of the following commands for further analysis:

fw ver

free -m
netstat -ni
enabled_blades
fwaccel stat
fw ctl multik stat
fw ctl affinity -l -r
fw ctl multik get_mode (R77.30) or fw ctl multik dynamic_dispatching get_mode (R80.10+)
cpstat os -f multi_cpu -o 1
cpconfig (the menu displayed by this command)

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Joshua_Hatter]

Also might be useful to have #fwaccel conns

[Thecoder]

fw ver
This is Check Point's software version R80.10 - Build 056
[Expert@fw1:0]# free -m
total used free shared buffers cached
Mem: 64207 35142 29064 0 903 12673
-/+ buffers/cache: 21565 42641
Swap: 32765 0 32765
[Expert@fw1:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 118199136 0 0 0 1672154958 0 0 0 BMRU
bond100 1500 0 125776735 0 0 0 1903103776 0 0 0 BMmRU
eth1-01 1500 0 6990601996 0 0 0 7257672547 0 0 0 BMRU
eth1-02 1500 0 2708488932 0 0 0 2616334278 0 0 0 BMRU
eth2-01 1500 0 5308103110 0 127382 127382 5964648490 0 0 0 BMRU
eth2-02 1500 0 4477011 0 0 0 4658400 0 0 0 BMRU
eth3-01 1500 0 0 0 0 0 0 0 0 0 BMU
eth3-03 1500 0 303419 0 0 0 299866 0 0 0 BMsRU
eth3-04 1500 0 125473320 0 0 0 1902803949 0 0 0 BMsRU
lo 16436 0 17779742 0 0 0 17779742 0 0 0 LRU
[Expert@fw1:0]# enabled_blades
fw vpn cvpn urlf av appi ips identityServer SSL_INSPECT anti_bot ThreatEmulation mon vpn
[Expert@fw1:0]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
NMR Templates : enabled
NMT Templates : enabled

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
McastRoutingV2, NMR, NMT, NAT64, GTPAcceleration,
SCTPAcceleration
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Hyper-Threading
(10) Disable Check Point SecureXL
(11) Check Point CoreXL
(12) Automatic start of Check Point Products

(13) Exit

cpstat os -f multi_cpu -o 1

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 1| 13| 86| 14| ?| 74156|
| 2| 0| 16| 84| 16| ?| 74164|
| 3| 25| 49| 26| 74| ?| 74169|
| 4| 20| 50| 30| 70| ?| 74175|
| 5| 19| 51| 30| 70| ?| 74180|
| 6| 21| 50| 29| 71| ?| 74187|
| 7| 27| 42| 31| 69| ?| 74194|
| 8| 15| 60| 25| 75| ?| 74199|
| 9| 16| 56| 28| 72| ?| 74204|
| 10| 22| 44| 34| 66| ?| 74210|
| 11| 23| 45| 32| 68| ?| 74216|
| 12| 25| 37| 38| 62| ?| 74222|
| 13| 27| 37| 36| 64| ?| 74226|
| 14| 22| 47| 31| 69| ?| 74232|
| 15| 19| 48| 33| 67| ?| 74240|
| 16| 24| 41| 35| 65| ?| 74246|
| 17| 5| 5| 90| 10| ?| 74252|
| 18| 16| 11| 74| 26| ?| 74255|
| 19| 22| 49| 29| 71| ?| 74261|
| 20| 20| 52| 27| 73| ?| 74266|
| 21| 14| 64| 22| 78| ?| 74272|
| 22| 23| 46| 31| 69| ?| 74276|
| 23| 24| 45| 32| 68| ?| 74281|
| 24| 23| 48| 30| 70| ?| 74287|
| 25| 18| 55| 27| 73| ?| 74292|
| 26| 22| 45| 33| 67| ?| 74299|
| 27| 23| 43| 33| 67| ?| 74306|
| 28| 26| 40| 34| 66| ?| 74313|
| 29| 24| 46| 30| 70| ?| 74321|
| 30| 22| 45| 33| 67| ?| 74325|
| 31| 25| 40| 35| 65| ?| 74331|
| 32| 21| 46| 33| 67| ?| 74335|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 1| 13| 86| 14| ?| 74156|
| 2| 0| 16| 84| 16| ?| 74164|
| 3| 25| 49| 26| 74| ?| 74169|
| 4| 20| 50| 30| 70| ?| 74175|
| 5| 19| 51| 30| 70| ?| 74180|
| 6| 21| 50| 29| 71| ?| 74187|
| 7| 27| 42| 31| 69| ?| 74194|
| 8| 15| 60| 25| 75| ?| 74199|
| 9| 16| 56| 28| 72| ?| 74204|
| 10| 22| 44| 34| 66| ?| 74210|
| 11| 23| 45| 32| 68| ?| 74216|
| 12| 25| 37| 38| 62| ?| 74222|
| 13| 27| 37| 36| 64| ?| 74226|
| 14| 22| 47| 31| 69| ?| 74232|
| 15| 19| 48| 33| 67| ?| 74240|
| 16| 24| 41| 35| 65| ?| 74246|
| 17| 5| 5| 90| 10| ?| 74252|
| 18| 16| 11| 74| 26| ?| 74255|
| 19| 22| 49| 29| 71| ?| 74261|
| 20| 20| 52| 27| 73| ?| 74266|
| 21| 14| 64| 22| 78| ?| 74272|
| 22| 23| 46| 31| 69| ?| 74276|
| 23| 24| 45| 32| 68| ?| 74281|
| 24| 23| 48| 30| 70| ?| 74287|
| 25| 18| 55| 27| 73| ?| 74292|
| 26| 22| 45| 33| 67| ?| 74299|
| 27| 23| 43| 33| 67| ?| 74306|
| 28| 26| 40| 34| 66| ?| 74313|
| 29| 24| 46| 30| 70| ?| 74321|
| 30| 22| 45| 33| 67| ?| 74325|
| 31| 25| 40| 35| 65| ?| 74331|
| 32| 21| 46| 33| 67| ?| 74335|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 13| 87| 13| ?| 54603|
| 2| 0| 15| 85| 15| ?| 54608|
| 3| 20| 52| 27| 73| ?| 54612|
| 4| 25| 49| 26| 74| ?| 54617|
| 5| 23| 47| 30| 70| ?| 54620|
| 6| 23| 46| 31| 69| ?| 54624|
| 7| 22| 51| 26| 74| ?| 54629|
| 8| 26| 44| 30| 70| ?| 54635|
| 9| 25| 46| 30| 70| ?| 54641|
| 10| 17| 56| 27| 73| ?| 54647|
| 11| 26| 43| 32| 68| ?| 54654|
| 12| 27| 41| 32| 68| ?| 54660|
| 13| 27| 45| 29| 71| ?| 54664|
| 14| 23| 50| 27| 73| ?| 54669|
| 15| 19| 53| 28| 72| ?| 54673|
| 16| 15| 63| 21| 79| ?| 54679|
| 17| 0| 2| 98| 2| ?| 54684|
| 18| 0| 3| 97| 3| ?| 54690|
| 19| 18| 55| 27| 73| ?| 54694|
| 20| 23| 48| 29| 71| ?| 54700|
| 21| 12| 67| 20| 80| ?| 54706|
| 22| 31| 32| 37| 63| ?| 54710|
| 23| 27| 37| 36| 64| ?| 54714|
| 24| 21| 51| 28| 72| ?| 54719|
| 25| 18| 54| 27| 73| ?| 54724|
| 26| 22| 52| 26| 74| ?| 54730|
| 27| 27| 41| 32| 68| ?| 54737|
| 28| 23| 47| 30| 70| ?| 54742|
| 29| 30| 40| 30| 70| ?| 54744|
| 30| 25| 43| 32| 68| ?| 54748|
| 31| 25| 47| 28| 72| ?| 54752|
| 32| 22| 50| 29| 71| ?| 54756|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 13| 87| 13| ?| 54603|
| 2| 0| 15| 85| 15| ?| 54608|
| 3| 20| 52| 27| 73| ?| 54612|
| 4| 25| 49| 26| 74| ?| 54617|
| 5| 23| 47| 30| 70| ?| 54620|
| 6| 23| 46| 31| 69| ?| 54624|
| 7| 22| 51| 26| 74| ?| 54629|
| 8| 26| 44| 30| 70| ?| 54635|
| 9| 25| 46| 30| 70| ?| 54641|
| 10| 17| 56| 27| 73| ?| 54647|
| 11| 26| 43| 32| 68| ?| 54654|
| 12| 27| 41| 32| 68| ?| 54660|
| 13| 27| 45| 29| 71| ?| 54664|
| 14| 23| 50| 27| 73| ?| 54669|
| 15| 19| 53| 28| 72| ?| 54673|
| 16| 15| 63| 21| 79| ?| 54679|
| 17| 0| 2| 98| 2| ?| 54684|
| 18| 0| 3| 97| 3| ?| 54690|
| 19| 18| 55| 27| 73| ?| 54694|
| 20| 23| 48| 29| 71| ?| 54700|
| 21| 12| 67| 20| 80| ?| 54706|
| 22| 31| 32| 37| 63| ?| 54710|
| 23| 27| 37| 36| 64| ?| 54714|
| 24| 21| 51| 28| 72| ?| 54719|
| 25| 18| 54| 27| 73| ?| 54724|
| 26| 22| 52| 26| 74| ?| 54730|
| 27| 27| 41| 32| 68| ?| 54737|
| 28| 23| 47| 30| 70| ?| 54742|
| 29| 30| 40| 30| 70| ?| 54744|
| 30| 25| 43| 32| 68| ?| 54748|
| 31| 25| 47| 28| 72| ?| 54752|
| 32| 22| 50| 29| 71| ?| 54756|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 13| 87| 13| ?| 25536|
| 2| 0| 14| 86| 14| ?| 25541|
| 3| 21| 49| 29| 71| ?| 25545|
| 4| 21| 46| 33| 67| ?| 25549|
| 5| 25| 41| 35| 65| ?| 25551|
| 6| 23| 49| 28| 72| ?| 25554|
| 7| 24| 44| 32| 68| ?| 25559|
| 8| 21| 51| 28| 72| ?| 25564|
| 9| 18| 52| 30| 70| ?| 25569|
| 10| 25| 43| 32| 68| ?| 25576|
| 11| 25| 40| 36| 64| ?| 25580|
| 12| 24| 43| 33| 67| ?| 25587|
| 13| 20| 45| 35| 65| ?| 25591|
| 14| 19| 53| 27| 73| ?| 25599|
| 15| 17| 52| 32| 68| ?| 25603|
| 16| 24| 39| 37| 63| ?| 25609|
| 17| 0| 2| 98| 2| ?| 25616|
| 18| 0| 1| 99| 1| ?| 25623|
| 19| 24| 44| 32| 68| ?| 25631|
| 20| 20| 50| 30| 70| ?| 25639|
| 21| 22| 51| 27| 73| ?| 25646|
| 22| 26| 39| 36| 64| ?| 25652|
| 23| 23| 42| 36| 64| ?| 25657|
| 24| 23| 45| 32| 68| ?| 25662|
| 25| 17| 51| 32| 68| ?| 25667|
| 26| 21| 41| 38| 62| ?| 25673|
| 27| 25| 40| 35| 65| ?| 25679|
| 28| 23| 41| 36| 64| ?| 25686|
| 29| 25| 41| 35| 65| ?| 25692|
| 30| 19| 45| 36| 64| ?| 25700|
| 31| 18| 53| 30| 70| ?| 25706|
| 32| 24| 42| 34| 66| ?| 25710|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 0| 13| 87| 13| ?| 25536|
| 2| 0| 14| 86| 14| ?| 25541|
| 3| 21| 49| 29| 71| ?| 25545|
| 4| 21| 46| 33| 67| ?| 25549|
| 5| 25| 41| 35| 65| ?| 25551|
| 6| 23| 49| 28| 72| ?| 25554|
| 7| 24| 44| 32| 68| ?| 25559|
| 8| 21| 51| 28| 72| ?| 25564|
| 9| 18| 52| 30| 70| ?| 25569|
| 10| 25| 43| 32| 68| ?| 25576|
| 11| 25| 40| 36| 64| ?| 25580|
| 12| 24| 43| 33| 67| ?| 25587|
| 13| 20| 45| 35| 65| ?| 25591|
| 14| 19| 53| 27| 73| ?| 25599|
| 15| 17| 52| 32| 68| ?| 25603|
| 16| 24| 39| 37| 63| ?| 25609|
| 17| 0| 2| 98| 2| ?| 25616|
| 18| 0| 1| 99| 1| ?| 25623|
| 19| 24| 44| 32| 68| ?| 25631|
| 20| 20| 50| 30| 70| ?| 25639|
| 21| 22| 51| 27| 73| ?| 25646|
| 22| 26| 39| 36| 64| ?| 25652|
| 23| 23| 42| 36| 64| ?| 25657|
| 24| 23| 45| 32| 68| ?| 25662|
| 25| 17| 51| 32| 68| ?| 25667|
| 26| 21| 41| 38| 62| ?| 25673|
| 27| 25| 40| 35| 65| ?| 25679|
| 28| 23| 41| 36| 64| ?| 25686|
| 29| 25| 41| 35| 65| ?| 25692|
| 30| 19| 45| 36| 64| ?| 25700|
| 31| 18| 53| 30| 70| ?| 25706|
| 32| 24| 42| 34| 66| ?| 25710|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 9| 20| 71| 29| ?| 57968|
| 2| 5| 15| 80| 20| ?| 57977|
| 3| 26| 47| 27| 73| ?| 57984|
| 4| 29| 41| 31| 69| ?| 57991|
| 5| 13| 65| 22| 78| ?| 57999|
| 6| 28| 44| 29| 71| ?| 58004|
| 7| 20| 55| 25| 75| ?| 58009|
| 8| 19| 54| 28| 72| ?| 58017|
| 9| 20| 51| 29| 71| ?| 58027|
| 10| 19| 50| 30| 70| ?| 58034|
| 11| 22| 44| 35| 65| ?| 58044|
| 12| 23| 39| 39| 61| ?| 58052|
| 13| 21| 48| 32| 68| ?| 58059|
| 14| 25| 42| 33| 67| ?| 58066|
| 15| 17| 56| 27| 73| ?| 58073|
| 16| 20| 52| 27| 73| ?| 58078|
| 17| 1| 5| 95| 5| ?| 58082|
| 18| 0| 7| 93| 7| ?| 58089|
| 19| 24| 48| 28| 72| ?| 58097|
| 20| 23| 49| 28| 72| ?| 58103|
| 21| 19| 49| 32| 68| ?| 58110|
| 22| 28| 41| 31| 69| ?| 58119|
| 23| 26| 43| 31| 69| ?| 58126|
| 24| 25| 44| 32| 68| ?| 58133|
| 25| 22| 48| 31| 69| ?| 58138|
| 26| 20| 51| 28| 72| ?| 58143|
| 27| 24| 43| 34| 66| ?| 58149|
| 28| 21| 47| 32| 68| ?| 58156|
| 29| 24| 42| 35| 65| ?| 58162|
| 30| 22| 47| 31| 69| ?| 58171|
| 31| 17| 52| 30| 70| ?| 58178|
| 32| 19| 49| 33| 67| ?| 58184|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 9| 20| 71| 29| ?| 57968|
| 2| 5| 15| 80| 20| ?| 57977|
| 3| 26| 47| 27| 73| ?| 57984|
| 4| 29| 41| 31| 69| ?| 57991|
| 5| 13| 65| 22| 78| ?| 57999|
| 6| 28| 44| 29| 71| ?| 58004|
| 7| 20| 55| 25| 75| ?| 58009|
| 8| 19| 54| 28| 72| ?| 58017|
| 9| 20| 51| 29| 71| ?| 58027|
| 10| 19| 50| 30| 70| ?| 58034|
| 11| 22| 44| 35| 65| ?| 58044|
| 12| 23| 39| 39| 61| ?| 58052|
| 13| 21| 48| 32| 68| ?| 58059|
| 14| 25| 42| 33| 67| ?| 58066|
| 15| 17| 56| 27| 73| ?| 58073|
| 16| 20| 52| 27| 73| ?| 58078|
| 17| 1| 5| 95| 5| ?| 58082|
| 18| 0| 7| 93| 7| ?| 58089|
| 19| 24| 48| 28| 72| ?| 58097|
| 20| 23| 49| 28| 72| ?| 58103|
| 21| 19| 49| 32| 68| ?| 58110|
| 22| 28| 41| 31| 69| ?| 58119|
| 23| 26| 43| 31| 69| ?| 58126|
| 24| 25| 44| 32| 68| ?| 58133|
| 25| 22| 48| 31| 69| ?| 58138|
| 26| 20| 51| 28| 72| ?| 58143|
| 27| 24| 43| 34| 66| ?| 58149|
| 28| 21| 47| 32| 68| ?| 58156|
| 29| 24| 42| 35| 65| ?| 58162|
| 30| 22| 47| 31| 69| ?| 58171|
| 31| 17| 52| 30| 70| ?| 58178|
| 32| 19| 49| 33| 67| ?| 58184|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 36| 33| 30| 70| ?| 61929|
| 2| 63| 29| 8| 92| ?| 61936|
| 3| 26| 48| 27| 73| ?| 61942|
| 4| 27| 46| 27| 73| ?| 61949|
| 5| 27| 45| 28| 72| ?| 61953|
| 6| 19| 58| 24| 76| ?| 61959|
| 7| 24| 52| 24| 76| ?| 61965|
| 8| 25| 49| 26| 74| ?| 61972|
| 9| 21| 52| 27| 73| ?| 61978|
| 10| 21| 53| 26| 74| ?| 61982|
| 11| 13| 68| 19| 81| ?| 61989|
| 12| 29| 43| 29| 71| ?| 61993|
| 13| 27| 44| 30| 70| ?| 61999|
| 14| 24| 46| 30| 70| ?| 62006|
| 15| 23| 49| 28| 72| ?| 62012|
| 16| 19| 55| 26| 74| ?| 62019|
| 17| 26| 28| 46| 54| ?| 62024|
| 18| 26| 24| 50| 50| ?| 62031|
| 19| 30| 49| 21| 79| ?| 62036|
| 20| 28| 45| 27| 73| ?| 62042|
| 21| 27| 47| 26| 74| ?| 62048|
| 22| 25| 46| 29| 71| ?| 62055|
| 23| 26| 46| 28| 72| ?| 62061|
| 24| 23| 55| 22| 78| ?| 62068|
| 25| 26| 45| 29| 71| ?| 62075|
| 26| 23| 51| 26| 74| ?| 62082|
| 27| 19| 45| 36| 64| ?| 62089|
| 28| 29| 43| 28| 72| ?| 62096|
| 29| 29| 41| 31| 69| ?| 62102|
| 30| 18| 57| 25| 75| ?| 62107|
| 31| 26| 42| 32| 68| ?| 62110|
| 32| 19| 52| 29| 71| ?| 62115|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 36| 33| 30| 70| ?| 61929|
| 2| 63| 29| 8| 92| ?| 61936|
| 3| 26| 48| 27| 73| ?| 61942|
| 4| 27| 46| 27| 73| ?| 61949|
| 5| 27| 45| 28| 72| ?| 61953|
| 6| 19| 58| 24| 76| ?| 61959|
| 7| 24| 52| 24| 76| ?| 61965|
| 8| 25| 49| 26| 74| ?| 61972|
| 9| 21| 52| 27| 73| ?| 61978|
| 10| 21| 53| 26| 74| ?| 61982|
| 11| 13| 68| 19| 81| ?| 61989|
| 12| 29| 43| 29| 71| ?| 61993|
| 13| 27| 44| 30| 70| ?| 61999|
| 14| 24| 46| 30| 70| ?| 62006|
| 15| 23| 49| 28| 72| ?| 62012|
| 16| 19| 55| 26| 74| ?| 62019|
| 17| 26| 28| 46| 54| ?| 62024|
| 18| 26| 24| 50| 50| ?| 62031|
| 19| 30| 49| 21| 79| ?| 62036|
| 20| 28| 45| 27| 73| ?| 62042|
| 21| 27| 47| 26| 74| ?| 62048|
| 22| 25| 46| 29| 71| ?| 62055|
| 23| 26| 46| 28| 72| ?| 62061|
| 24| 23| 55| 22| 78| ?| 62068|
| 25| 26| 45| 29| 71| ?| 62075|
| 26| 23| 51| 26| 74| ?| 62082|
| 27| 19| 45| 36| 64| ?| 62089|
| 28| 29| 43| 28| 72| ?| 62096|
| 29| 29| 41| 31| 69| ?| 62102|
| 30| 18| 57| 25| 75| ?| 62107|
| 31| 26| 42| 32| 68| ?| 62110|
| 32| 19| 52| 29| 71| ?| 62115|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 14| 44| 42| 58| ?| 55755|
| 2| 2| 24| 73| 27| ?| 55761|
| 3| 25| 40| 35| 65| ?| 55765|
| 4| 18| 47| 34| 66| ?| 55770|
| 5| 21| 44| 35| 65| ?| 55774|
| 6| 22| 39| 39| 61| ?| 55779|
| 7| 19| 51| 30| 70| ?| 55785|
| 8| 23| 39| 39| 61| ?| 55789|
| 9| 25| 37| 38| 62| ?| 55791|
| 10| 16| 53| 31| 69| ?| 55797|
| 11| 26| 34| 40| 60| ?| 55803|
| 12| 24| 35| 40| 60| ?| 55808|
| 13| 20| 40| 40| 60| ?| 55813|
| 14| 21| 40| 39| 61| ?| 55818|
| 15| 19| 41| 40| 60| ?| 55823|
| 16| 20| 42| 37| 63| ?| 55830|
| 17| 23| 31| 46| 54| ?| 55835|
| 18| 92| 8| 0| 100| ?| 55839|
| 19| 22| 45| 34| 66| ?| 55844|
| 20| 18| 50| 32| 68| ?| 55851|
| 21| 23| 43| 34| 66| ?| 55854|
| 22| 29| 35| 36| 64| ?| 55858|
| 23| 25| 34| 41| 59| ?| 55861|
| 24| 22| 45| 32| 68| ?| 55866|
| 25| 25| 38| 37| 63| ?| 55870|
| 26| 23| 41| 36| 64| ?| 55874|
| 27| 24| 39| 37| 63| ?| 55879|
| 28| 26| 35| 39| 61| ?| 55884|
| 29| 22| 41| 38| 62| ?| 55889|
| 30| 19| 46| 35| 65| ?| 55892|
| 31| 15| 52| 33| 67| ?| 55898|
| 32| 19| 44| 38| 62| ?| 55902|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 14| 44| 42| 58| ?| 55755|
| 2| 2| 24| 73| 27| ?| 55761|
| 3| 25| 40| 35| 65| ?| 55765|
| 4| 18| 47| 34| 66| ?| 55770|
| 5| 21| 44| 35| 65| ?| 55774|
| 6| 22| 39| 39| 61| ?| 55779|
| 7| 19| 51| 30| 70| ?| 55785|
| 8| 23| 39| 39| 61| ?| 55789|
| 9| 25| 37| 38| 62| ?| 55791|
| 10| 16| 53| 31| 69| ?| 55797|
| 11| 26| 34| 40| 60| ?| 55803|
| 12| 24| 35| 40| 60| ?| 55808|
| 13| 20| 40| 40| 60| ?| 55813|
| 14| 21| 40| 39| 61| ?| 55818|
| 15| 19| 41| 40| 60| ?| 55823|
| 16| 20| 42| 37| 63| ?| 55830|
| 17| 23| 31| 46| 54| ?| 55835|
| 18| 92| 8| 0| 100| ?| 55839|
| 19| 22| 45| 34| 66| ?| 55844|
| 20| 18| 50| 32| 68| ?| 55851|
| 21| 23| 43| 34| 66| ?| 55854|
| 22| 29| 35| 36| 64| ?| 55858|
| 23| 25| 34| 41| 59| ?| 55861|
| 24| 22| 45| 32| 68| ?| 55866|
| 25| 25| 38| 37| 63| ?| 55870|
| 26| 23| 41| 36| 64| ?| 55874|
| 27| 24| 39| 37| 63| ?| 55879|
| 28| 26| 35| 39| 61| ?| 55884|
| 29| 22| 41| 38| 62| ?| 55889|
| 30| 19| 46| 35| 65| ?| 55892|
| 31| 15| 52| 33| 67| ?| 55898|
| 32| 19| 44| 38| 62| ?| 55902|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 25| 41| 34| 66| ?| 58551|
| 2| 13| 24| 64| 36| ?| 58557|
| 3| 22| 39| 40| 60| ?| 58561|
| 4| 15| 46| 39| 61| ?| 58564|
| 5| 19| 39| 42| 58| ?| 58567|
| 6| 12| 52| 37| 63| ?| 58572|
| 7| 17| 48| 35| 65| ?| 58575|
| 8| 18| 45| 37| 63| ?| 58579|
| 9| 26| 38| 36| 64| ?| 58583|
| 10| 20| 50| 29| 71| ?| 58587|
| 11| 26| 39| 35| 65| ?| 58593|
| 12| 26| 38| 36| 64| ?| 58600|
| 13| 26| 37| 37| 63| ?| 58605|
| 14| 25| 36| 40| 60| ?| 58609|
| 15| 21| 47| 32| 68| ?| 58613|
| 16| 24| 38| 38| 62| ?| 58618|
| 17| 29| 31| 40| 60| ?| 58622|
| 18| 91| 9| 0| 100| ?| 58626|
| 19| 20| 42| 38| 62| ?| 58632|
| 20| 14| 53| 34| 66| ?| 58638|
| 21| 19| 41| 41| 59| ?| 58642|
| 22| 16| 49| 36| 64| ?| 58646|
| 23| 23| 34| 44| 56| ?| 58650|
| 24| 20| 38| 42| 58| ?| 58655|
| 25| 27| 39| 34| 66| ?| 58659|
| 26| 21| 44| 35| 65| ?| 58664|
| 27| 24| 38| 37| 63| ?| 58668|
| 28| 25| 43| 33| 67| ?| 58672|
| 29| 26| 36| 38| 62| ?| 58675|
| 30| 23| 44| 33| 67| ?| 58680|
| 31| 28| 36| 36| 64| ?| 58684|
| 32| 25| 41| 34| 66| ?| 58690|

[Timothy_Hall]

Networker Networker‌ you didn't post the output of fw ctl affinity -l -r, but based on the CPU behavior it would appear you have a 2/30 split of SND/IRQ cores to Firewall Workers.  The low RX-DRP rate reported by netstat seems to indicate that the SND/IRQ cores are servicing the network ring buffers adequately.

However I'm noticing a lot of user/process space CPU usage on the Firewall Worker cores and given you have HTTPS Inspection enabled, I'd guess that the wstlsd process assigned to each worker core in the cause.  All traffic that is subject to HTTPS Inspection will go F2F.  Selecting "Categorize HTTPS Sites" without enabling full-fledged HTTPS Inspection will also cause this to a lesser degree.  Please review your HTTPS Inspection policy, you should NEVER use the object "Any" in the source or destination fields as it will pull all kinds of traffic into F2F.

The only other blade you have enabled that can cause major trips up into process space through F2F is Threat Emulation via the dplu and ted processes.  I'd say your next step is to identify via top what processes are eating process/user space CPU time on the worker cores (2-31) and that should get you going in the right direction.

In regards to the zero templating rate (Accelerated conns/Total conns) noted by another poster, this is caused by enabling the anti-bot blade and is noted in the second edition of my book.  Also if you have anything other than "Firewall" selected in your first policy layer (Network) that will also crater the SecureXL templating rate to zero.  Doesn't matter all that much performance-wise though due to the new Column-based rulebase matching feature in R80.10.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Thecoder]

hi Timm

i attached new log file about your requests. https inspection active on fw. i saw that there are many wstlsd process in result of top command. And we have two access policy layers:security and application. firewall layer is only opened in security layer. 

[Thecoder]

i exchanged https inspection rules(sources=any->all network range destination=ayn->internet)

but amount of traffic of f2f didnt change.

[Timothy_Hall]

Networker Networker‌ after looking at your revised file you have a 4/28 split.  Pretty sure your high F2F is caused by HTTPS Inspection and the associated process space trip up to wstlsd, unfortunately there is very little you can do about it other than optimizing your HTTPS Inspection policy as mentioned.  pkxld is running so you definitely have Gaia executing in 64-bit mode and are taking full advantage of the various processor extensions and AES-NI.  The only way to conclusively determine if HTTPS Inspection is causing the high F2F is to disable it on the gateway/cluster temporarily, run fwaccel stats -r, wait a few minutes then run fwaccel stats -s again.

The zero SecureXL templating rate (Accelerated conns/Total conns) is being caused by anti-bot and has nothing to do with the high F2F percentage.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Timothy_Hall]

I almost put this in my reply above, but wasn't sure how much was public yet about R80.20.  But now that the R80.20 has entered public EA and the release notes are semi-public, I can state that there is some relief coming for the high CPU impact of HTTPS Inspection, even if your firewall does not have the new Falcon accelerator card (but of course the card will *really* help a lot).  Still going through all the new R80.20 stuff but looks quite promising thus far.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Jelle_Hazenberg]

Hi,

Since when did you have the performance problems (Is there something changed in the environment)?

Which blades are there enabled? (enter enabled_blades in expert mode)

Show the ouput of the "sar" command

Look at the output of fwaccel conns and compare them to the following rules:

• The first packet of any new TCP or UDP session, unless there is a template
• Connections destined to or from the Security Gateway
• Connections that require Security servers
• Connections that have a Handler (ICMP, FTP,)
• Some IPS features (IP, ID, TTL)
• Multicast packets

Best luck with the troubleshooting!

Sounds like an interesting but yet irritating problem...

Greetz,

Jelle

[Thecoder]

we opened strict inspection and ips profile. enabled_blades commands applied in expert mode

i attached sar log file to original post.

[Martin_Valenta]

Well if you have enabled NGTP blades you cannot expect much to be accelarated. All NGTP traffic goes to PXL. You can verify that once you disable those blades, all will go to Accelerated path

[Thecoder]

i disabled ips and threat prevention for test. but the result not changed. not accelerated.

[Martin_Valenta]

and with expert command: fwaccel off/on ?

[Thecoder]

yes

[Martin_Valenta]

fwaccel stat - shows what?

[Thecoder]

result of fwaccel stat not changed. accelerated conn=%0

[Martin_Valenta]

That would mean that traffic passed throw gateway cannot be accelarated. What kind of services are passing firewall?

[G_W_Albrecht]

sk98348: Best Practices - Security Gateway Performance

[Thecoder]

yes, traffic passed throw gateway but not accelarated. our top protocol http,https,vpn,imap,pop3,sql,kerboros,naspter,dns,echo-request

[Markus_Genser]

Do you use ISP redundancy with load sharing?

[Danny_Olson]

Yeah, you can only accelerate traffic, or inspect it but not both. I have similar stats (below), and no traffic being accelerated, but we average 50MB of throughput going though the box (on a 12600), so we can get away with it. I have FW, VPN, APP/URLF, IA, AV, AB, IPS. What amount of throughput are you seeing though the box? Are you seeing the performance issues all the time, or just certain times? Do you have high connection rate? Dynamic dispatcher address a lot of issues. Are you seeing drops/overruns?  Maybe your SND/Dispatcher ratio could use some adjusting? 

Accelerated conns/Total conns : 49/37356 (0%)
Accelerated pkts/Total pkts : 3376791/195771942 (1%)
F2Fed pkts/Total pkts : 11224710/195771942 (5%)
PXL pkts/Total pkts : 181170441/195771942 (92%)
QXL pkts/Total pkts : 0/195771942 (0%)

[Thecoder]

our throughput is between 800mb-1gb.we are seeing the performance issues in working time. we have average 100000 connection. i appiled corexl. and allocated cpu for important intarfaces

[PhoneBoy]

Even with all the blades enabled, you shouldn't see the amount of F2F traffic you're seeing.

[Thecoder]

probably we found the problem. we realized that there were too many /connect traffics. and we disabled to redirect to the captive portal. And The cpu down from %80 to %45