topic Re: SecureXL Accelerated path at 20% R77.30 in General Topics

SecureXL Accelerated path at 20% R77.30

Terry — Tue, 11 Feb 2020 22:37:24 GMT

I cannot find the reason that our CP firewalls only accelerate 20% of our traffic. We run VSX, CoreXL is enabled. I have reviewed the Max Power V2 guide for tuning. I'm at a loss. Any ideas?

[Expert@PROD-B:10]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

[Expert@PROD-B:10]# fwaccel stats -s
Accelerated conns/Total conns : 16327/71523 (22%)
Delayed conns/(Accelerated conns + PXL conns) : 602/68790 (0%)
Accelerated pkts/Total pkts : 63124135/190394161 (33%)
F2Fed pkts/Total pkts : 20142622/190394161 (10%)
PXL pkts/Total pkts : 107127404/190394161 (56%)
QXL pkts/Total pkts : 0/190394161 (0%)

[Expert@PROD-B:10]# grep -c ^processor /proc/cpuinfo && /sbin/cpuinfo
6
HyperThreading=disabled

[Expert@PROD-B:10]# vsenv 0
Context is set to Virtual Device PROD-B (ID 0).
[Expert@PROD-B:0]# fw ctl affinity -l -r
CPU 0: Sync Mgmt
CPU 1: eth1-04
CPU 2:
CPU 3:
CPU 4:
CPU 5:
All:

[Expert@PROD-B:10]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1-04.9 1500 0 23492538 0 0 0 20331997 0 0 0 BMRU
eth1-04.86 1500 0 2087760167 0 0 0 3853436307 0 0 0 BMRU
eth1-04.95 1500 0 196787791 0 0 0 85395544 0 0 0 BMRU
eth1-04.803 1500 0 156942065 0 0 0 162015933 0 0 0 BMRU
eth1-04.825 1500 0 16917067 0 0 0 12161382 0 0 0 BMRU
lo10 16436 0 159309633 0 0 0 159309633 0 0 0 LRU
wrp640 1500 0 38286940 0 0 0 6429621 0 0 0 BMRU
wrp641 1600 0 11803588 0 0 0 42741043 0 0 0 BMRU
wrp642 1500 0 10795355 0 0 0 4582615 0 0 0 BMRU
wrp643 1500 0 16293750 0 0 0 21117665 0 0 0 BMRU
wrp644 1500 0 7055301 0 0 0 2288196 0 0 0 BMRU
wrp646 1500 0 5620843 0 0 0 2833859912 0 0 0 BMRU
wrp648 1500 0 1776562 0 0 0 2254472498 0 0 0 BMRU
wrp649 1500 0 7537011 0 0 0 4227601 0 0 0 BMRU
wrp650 1500 0 6117743 0 0 0 826096629 0 0 0 BMRU
wrp651 1500 0 7146004 0 0 0 3435033 0 0 0 BMRU
wrp652 1500 0 7692419 0 0 0 14455124 0 0 0 BMRU
wrp653 1500 0 7238752 0 0 0 3655612 0 0 0 BMRU
wrp654 1500 0 26167255 0 0 0 456417780 0 0 0 BMRU

[Expert@PROD-B:10]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 2-5 | 25185 | 59889
1 | Yes | 2-5 | 15316 | 38149
2 | Yes | 2-5 | 15463 | 49210
3 | Yes | 2-5 | 17479 | 56844

[Expert@PROD-B:10]# cpstat os -f multi_cpu -o 1 -c 5

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 8| 38| 54| 46| ?| 23016|
| 2| 8| 37| 55| 45| ?| 23016|
| 3| 20| 19| 61| 39| ?| 23016|
| 4| 23| 17| 60| 40| ?| 23016|
| 5| 20| 20| 60| 40| ?| 23016|
| 6| 20| 20| 60| 40| ?| 23016|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 10| 38| 53| 47| ?| 73050|
| 2| 11| 41| 49| 51| ?| 73051|
| 3| 20| 23| 56| 44| ?| 73052|
| 4| 28| 23| 49| 51| ?| 73054|
| 5| 18| 27| 56| 44| ?| 73054|
| 6| 23| 25| 52| 48| ?| 146113|
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 7| 36| 57| 43| ?| 74892|
| 2| 11| 41| 48| 52| ?| 74896|
| 3| 24| 24| 53| 47| ?| 149796|
| 4| 19| 22| 59| 41| ?| 149800|
| 5| 22| 26| 53| 47| ?| 74901|
| 6| 21| 28| 51| 49| ?| 74902|
---------------------------------------------------------------------------------

Re: SecureXL Accelerated path at 20% R77.30

Daniel_Schlifka — Tue, 11 Feb 2020 22:51:14 GMT

Can you provide the output of fwaccel stats (without -s param)?

Re: SecureXL Accelerated path at 20% R77.30

Timothy_Hall — Tue, 11 Feb 2020 22:57:15 GMT

Need to see output of enabled_blades please, glad you liked Max Power V2. 10% F2F is not horrible, and depending on what blades you have enabled, 54% PXL may not indicate a problem.

Re: SecureXL Accelerated path at 20% R77.30

Terry — Wed, 12 Feb 2020 14:21:32 GMT

I love your book. A must-have for any Checkpoint Engineer.

Here is the output.

[Expert@PROD-B:10]# enabled_blades
fw vpn cvpn urlf appi identityServer

Re: SecureXL Accelerated path at 20% R77.30

Terry — Wed, 12 Feb 2020 14:17:39 GMT

[Expert@PROD-B:10]# fwaccel stats
Name Value Name Value
-------------------- --------------- -------------------- ---------------

Accelerated Path
------------------------------------------------------------------------------
accel packets 81556 accel bytes 45729320
conns created 6805 conns deleted 32
C total conns 7131 C templates 354
C TCP conns 5585 C delayed TCP conns 10
C non TCP conns 1546 C delayed nonTCP con 0
conns from templates 547 temporary conns 175
nat conns 4355 dropped packets 146
dropped bytes 20071 nat templates 0
port alloc templates 0 conns from nat tmpl 0
port alloc conns 0 conns auto expired 0

Accelerated VPN Path
------------------------------------------------------------------------------
C crypt conns 421 enc bytes 14724672
dec bytes 2141520 ESP enc pkts 13827
ESP enc err 0 ESP dec pkts 11184
ESP dec err 0 ESP other err 0
AH enc pkts 0 AH enc err 0
AH dec pkts 0 AH dec err 0
AH other err 0 espudp enc pkts 0
espudp enc err 0 espudp dec pkts 0
espudp dec err 0 espudp other err 0

Medium Path
------------------------------------------------------------------------------
PXL packets 169762 PXL async packets 158254
PXL bytes 118811908 C PXL conns 4898
C PXL templates 167 PXL FF conns 48
PXL FF packets 12019 PXL FF bytes 10610774
PXL FF acks 4589

Accelerated QoS Path
------------------------------------------------------------------------------
QXL packets 0 QXL async packets 0
QXL bytes 0 C QXL conns 0
C QXL templates 0

Firewall Path
------------------------------------------------------------------------------
F2F packets 35311 F2F bytes 20197052
C F2F conns 361 TCP violations 299
C partial conns 0 C anticipated conns 0
port alloc f2f 0

GTP
------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0

General
------------------------------------------------------------------------------
memory used 0 free memory 0
C used templates 161 pxl tmpl conns 368
C conns from tmpl 518 C non TCP F2F conns 134
C tcp handshake conn 298 C tcp established co 3703
C tcp closed conns 1584 C tcp f2f handshake 3
C tcp f2f establishe 192 C tcp f2f closed con 32
C tcp pxl handshake 269 C tcp pxl establishe 2519
C tcp pxl closed con 1316 outbound packets 81556
outbound pxl packets 169762 outbound f2f packets 37373
outbound bytes 47546675 outbound pxl bytes 121816875
outbound f2f bytes 22317085

(*) Statistics marked with C refer to current value, others refer to total value

Re: SecureXL Accelerated path at 20% R77.30

Timothy_Hall — Wed, 12 Feb 2020 14:40:34 GMT

Not seeing anything crazy bad in those outputs, it is APCL/URLF that is causing the traffic to be pulled into PXL which is expected. What you need to do is optimize your APCL/URLF policy to ensure that only the Internet-bound traffic is subject to APCL/URLF inspection and therefore PXL. For the R77.30 release specifically, check these things in your APCL/URLF policy:

1) Make sure that there is no APCL/URLF rule with "Any" in the Destination, you almost certainly should be using object Internet instead. As mentioned in my book, this assumes that your firewall topology is completely and correctly defined, wouldn't hurt to double-check it.

2) Avoid using Any in the Source field of your APCL/URLF rules as well, you should almost certainly be using some kind of network object that represents all your internal networks instead. Watch out for Access Role objects in the Source field that in their properties are set for "Any Networks" as these are not immediately obvious.

3) If you have the default "Any Internet Any Recognized Allow Log" rule at the bottom of your APCL/URLF policy, it is not necessary and can be deleted since the action of the implied cleanup rule is Accept for a APCL/URLF policy. Note that while doing so can substantially increase the amount of fully-accelerated traffic, it will eliminate logs for accepted applications previously matching this rule which will impact your Internet usage reporting. Worth a try though at least for testing.

4) If you've done all that and PXL is still unacceptable, try running fwaccel conns and look for connections that have the "S" flag showing. If they are connections from the inside to the Internet that is fine, but if you see connections between high-speed internal networks with S present you still have something wrong in your APCL/URLF policy.

Finally, remember that the PXL path is partially accelerated and not nearly as bad as the F2F path. In R80.20 and later much more traffic can be fully-accelerated due to the big changes in SecureXL.

Re: SecureXL Accelerated path at 20% R77.30

Terry — Wed, 12 Feb 2020 14:55:19 GMT

I ran through most of this. Very much appreciate the reinforcement. What do you think about whitelisting all traffic in our app/urlf if it matches destination port 443? HTTPS inspection is currently disabled. Will checkpoint try to inspect this traffic without inspection?

Re: SecureXL Accelerated path at 20% R77.30

Timothy_Hall — Wed, 12 Feb 2020 21:48:42 GMT

Sounds like you may have done all you can do as far as optimization. When you say whitelisting do you mean fast_accel: sk139772: SecureXL Fast Accelerator (sim fastaccel) for Non Scalable Platforms R77.30/R80.10

There will be some inspection of the whitelisted traffic but it will be limited to what SecureXL can do, which is...limited.

Re: SecureXL Accelerated path at 20% R77.30

PhoneBoy — Wed, 12 Feb 2020 22:56:25 GMT

URLF and APPI will definitely operate in PXL.
Perhaps you can optimize some of the traffic out of PXL by reordering or restructuring certain rules.

Re: SecureXL Accelerated path at 20% R77.30

Terry — Tue, 18 Feb 2020 14:03:19 GMT

Whoa...learn something new every day. Thank you for this. It should provide a significant throughput increase.

Re: SecureXL Accelerated path at 20% R77.30

Terry — Tue, 18 Feb 2020 14:04:29 GMT

I've tested several methods. I think we are as optimized as we can be. I appreciate the feedback!