https inspection performance issue - F2F traffic 8... - Page 2

andy_currigan · ‎2023-02-06

we have 2x open server in cluster XL r81 with a 4x cpu license

running https://openspeedtest.com/ (run on https) we notice an important drop in terms of performance, instead of 450-500 mbps that we get from a standard speedtest liko ookla that run on port 8080 the performance drops to max 150 mbps

we investigate the https inspection module but we do not understand the following behaviour.

1) if we create a rule in first position that bypass the entire https inspection for a specific host openspeedtest run at 450 mbps but if we insert the same host in a rule in position 5 that bypass the https inspection if you're member of a group of hosts the same host run at 150 mbps.

To note that in both case the url https://openspeedtest.com/ is bypassed due his categorization..

how is possible such a performance drop based on the https rule position?

why there's such a drop considering that this service is not even inspected?

we did some debug and we notice that lot's of traffic is goes through F2F below some usefull outputs.

any suggestions? tx

[Expert@checkpoint-1:0]# fwaccel stats -s
Accelerated conns/Total conns : 135/1570 (8%)
Accelerated pkts/Total pkts : 13101094681/73808428002 (17%)
F2Fed pkts/Total pkts : 60707333321/73808428002 (82%)
F2V pkts/Total pkts : 65943865/73808428002 (0%)
CPASXL pkts/Total pkts : 16454023/73808428002 (0%)
PSLXL pkts/Total pkts : 12372846626/73808428002 (16%)
CPAS pipeline pkts/Total pkts : 0/73808428002 (0%)
PSL pipeline pkts/Total pkts : 0/73808428002 (0%)
CPAS inline pkts/Total pkts : 0/73808428002 (0%)
PSL inline pkts/Total pkts : 0/73808428002 (0%)
QOS inbound pkts/Total pkts : 0/73808428002 (0%)
QOS outbound pkts/Total pkts : 0/73808428002 (0%)
Corrected pkts/Total pkts : 0/73808428002 (0%)

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
[Expert@checkpoint-1:0]#

[Expert@checkpoint-1:0]# fwaccel stats
Name Value Name Value
---------------------------- ------------ ---------------------------- ------------

Accelerated Path
--------------------------------------------------------------------------------------
accel packets 13105476237 accel bytes 9576220636092
outbound packets 13306779203 outbound bytes 9868235245852
conns created 65305430 conns deleted 65304144
C total conns 1286 C TCP conns 478
C non TCP conns 808 nat conns 31963228
dropped packets 2907232 dropped bytes 629465065
fragments received 1831431 fragments transmit 1126
fragments dropped 0 fragments expired 111439
IP options stripped 374709 IP options restored 115724
IP options dropped 0 corrs created 0
corrs deleted 0 C corrections 0
corrected packets 0 corrected bytes 0

Accelerated VPN Path
--------------------------------------------------------------------------------------
C crypt conns 115 enc bytes 201667504
dec bytes 682230912 ESP enc pkts 528421
ESP enc err 0 ESP dec pkts 708080
ESP dec err 0 ESP other err 0
espudp enc pkts 0 espudp enc err 0
espudp dec pkts 0 espudp dec err 0
espudp other err 0

Medium Streaming Path
--------------------------------------------------------------------------------------
CPASXL packets 16454121 PSLXL packets 12377039895
CPASXL async packets 16454023 PSLXL async packets 12376375499
CPASXL bytes 16508578013 PSLXL bytes 8937294264245
C CPASXL conns 3 C PSLXL conns 1157
CPASXL conns created 50783 PSLXL conns created 64697868
PXL FF conns 0 PXL FF packets 29416
PXL FF bytes 23581461 PXL FF acks 12056
PXL no conn drops 0

Pipeline Streaming Path
--------------------------------------------------------------------------------------
PSL Pipeline packets 0 PSL Pipeline bytes 0
CPAS Pipeline packets 0 CPAS Pipeline bytes 0

Inline Streaming Path
--------------------------------------------------------------------------------------
PSL Inline packets 0 PSL Inline bytes 0
CPAS Inline packets 0 CPAS Inline bytes 0

Buffer Path
--------------------------------------------------------------------------------------
Buffer path buffers 0 Buffer path bytes 0

TLS PARSER
--------------------------------------------------------------------------------------
RECORD INFO 0

TLS DECRYPT
--------------------------------------------------------------------------------------
TLS INSPECTION 0 TLS HANDSHAKE 0
TLS RECORD LAYER 0 TLS CRYPTO 0

HTTP DISP
--------------------------------------------------------------------------------------
ACTIVATE WS MAIN 0 EXEC NO HTTP CMI CONTEXT 0

WS LITE
--------------------------------------------------------------------------------------
WS TX COMPLETED 0 WS FORWARD TO MAIN 0
WS NOTIFY TIMEOUT 0 WS HANDLE EVENT 0
WS CHUNKED ERROR 0 WS GZIP EVENT 0
WS ADD MAC HEADER 0 WS IS STICKY ACTIVE 0
WS TIER1 JOB ERROR 0 WS TIER1 HAS MATCHES 0
CML MATCHES 0 TOTAL UPLOADED JOBS 0
TOTAL JOBS 0

ADVP
--------------------------------------------------------------------------------------
ADVP FORW TO MAIN 0 ADVP HOLD TIMEOUT 0

QoS Paths
--------------------------------------------------------------------------------------
QoS General Information:
------------------------
Total QoS Conns 0 QoS Classify Conns 0
QoS Classify flow 0 Reclassify QoS policy 0

FireWall QoS Path:
------------------
Enqueued IN packets 0 Enqueued OUT packets 0
Dequeued IN packets 0 Dequeued OUT packets 0
Enqueued IN bytes 0 Enqueued OUT bytes 0
Dequeued IN bytes 0 Dequeued OUT bytes 0

Accelerated QoS Path:
---------------------
Enqueued IN packets 0 Enqueued OUT packets 0
Dequeued IN packets 0 Dequeued OUT packets 0
Enqueued IN bytes 0 Enqueued OUT bytes 0
Dequeued IN bytes 0 Dequeued OUT bytes 0

Firewall Path
--------------------------------------------------------------------------------------
F2F packets 60736129970 F2F bytes 50320444505511
TCP violations 16 F2V conn match pkts 703989
F2V packets 65971694 F2V bytes 5570164544

GTP
--------------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0

General
--------------------------------------------------------------------------------------
memory used 103915120 C tcp handshake conns 14
C tcp established conns 454 C tcp closed conns 10
C tcp pxl handshake conns 14 C tcp pxl established conns 351
C tcp pxl closed conns 10 DNS DoR stats 291

(*) Statistics marked with C refer to current value, others refer to total value

[Expert@checkpoint-1:0]#

ragione del non accelerazione

[Expert@checkpoint-1:0]# fwaccel stats -p
F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
Pkt has IP options 374766 ICMP miss conn 249518305
TCP-SYN miss conn 215843802 TCP-other miss conn 29156577762
UDP miss conn 920603464 Other miss conn 5798272
VPN returned F2F 128716 Uni-directional viol 0
Possible spoof viol 0 TCP state viol 547
SCTP state affecting 0 Out if not def/accl 0
Bridge src=dst 0 Routing decision err 0
Sanity checks failed 0 Fwd to non-pivot 0
Broadcast/multicast 0 Cluster message 109434977
Cluster forward 635 Chain forwarding 0
F2V conn match pkts 705245 General reason 0
Route changes 0 VPN multicast traffic 0
GTP non-accelerated 0 Unresolved nexthop 38438
[Expert@checkpoint-1:0]# fwaccel stats -t
F2Fed bytes/Total bytes : 50329254032099/59906647972752 (84%)
F2V bytes/Total bytes : 5571144798/59906647972752 (0%)
Medium path bytes/Total bytes : 8954924374718/59906647972752 (14%)
Pipeline path bytes/Total bytes : 0/59906647972752 (0%)
Inline path bytes/Total bytes : 0/59906647972752 (0%)
Buffer path bytes/Total inline bytes: 0/0 (0%)

[Expert@checkpoint-1:0]# fwaccel stats -p
F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
Pkt has IP options 408329 ICMP miss conn 272317499
TCP-SYN miss conn 239969254 TCP-other miss conn 31278648253
UDP miss conn 1009197422 Other miss conn 6353935
VPN returned F2F 129009 Uni-directional viol 0
Possible spoof viol 0 TCP state viol 634
SCTP state affecting 0 Out if not def/accl 0
Bridge src=dst 0 Routing decision err 0
Sanity checks failed 0 Fwd to non-pivot 0
Broadcast/multicast 0 Cluster message 119931860
Cluster forward 635 Chain forwarding 0
F2V conn match pkts 1053535 General reason 0
Route changes 0 VPN multicast traffic 0
GTP non-accelerated 0 Unresolved nexthop 62227

[Expert@checkpoint-1:0]# fw ctl affinity -l -r
CPU 0:
CPU 1: fw_1 (active)
mpdaemon fwd rad lpd rtmd wsdnsd in.asessiond core_uploader cprid usrchkd vpnd in.acapd pepd pdpd cprid cpd
CPU 2:
CPU 3:
CPU 4:
CPU 5:
CPU 6:
CPU 7:
CPU 8:
CPU 9: fw_0 (active)
mpdaemon fwd rad lpd rtmd wsdnsd in.asessiond core_uploader cprid usrchkd vpnd in.acapd pepd pdpd cprid cpd
CPU 10:
CPU 11:
CPU 12:
CPU 13:
CPU 14:
CPU 15:
All:
The current license permits the use of CPUs 0, 1, 8, 9 only.
Interface eth4: has multi queue enabled
Interface eth5: has multi queue enabled
Interface eth0: has multi queue enabled
Interface eth6: has multi queue enabled
Interface eth3: has multi queue enabled
[Expert@checkpoint-1:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 9 | 5940 | 11921
1 | Yes | 1 | 6379 | 13536

[Expert@checkpoint-1:0]# enabled_blades
fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot mon

Timothy_Hall · ‎2023-02-06

As Chris said you must follow the rules order precisely as specified in those two links. The reason it is so important to have the rules in the correct order is due to the two-pass matching on the HTTPS Inspection policy. The first pass happens with the first packet of a new connection based on just IP addresses and ports. If a matching bypass rule can be found with this limited information (no domains/sites/categories allowed before any matching bypass rule here) active streaming is avoided and the connection can be passively streamed in the medium path or even fully accelerated by SecureXL.

If a matching Bypass cannot be found in the first pass, active streaming must be invoked to determine domain/site/category in the next few packets of the connection which incurs significant overhead, then the second pass occurs.

Are you sure you didn't run that fwaccel stats -s command on the standby member of your cluster? High F2F is expected there. HTTPS Inspection alone should not cause high F2F on the active member, something else is causing that, probably IPS. Try the fw tab -f -u -z -t connections command to see exclusively what connections are F2F and what they have in common. If that doesn't work try fw_streaming path slow (not sure on syntax there, typing all this on my phone). It looks like the main F2F violation causing the high F2F is "TCP other miss conn". Too generic so a debug must be run to find reason for F2F.

Everything I just posted is described and executed in labs (including the F2F debug) in my new live R81.20 Gateway Performance Optimization class.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com

CheckPointerXL · ‎2023-02-08

Hello Tim,

i've got list of F2F connections by executing fw tab -t connections -z (altough seems to display not only F2F because of no value under Not Offloaded Reason); how can i easily identify most impacting connections?

PS: fw_streaming path slow seems to be very fantasious, no way to guess correct syntax 😄

and

fw tab -f -u -z -t connections
Using cptfmt
Formatting table's data - this might take a while...

-z option must be used with connections table

Timothy_Hall · ‎2023-02-08

To find the most impacting connections run the secret hcp Threat Prevention reports I mentioned elsewhere in this thread, as it identifies top elephant flows subject to TP inspection.

As far as fw_streaming it looks like the "path slow" argument only works in R81.20+.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com

the_rock · ‎2023-02-08

I read all the comments and responses given and here is all I can say. I know lots of people advise when it comes to https inspection to use rule any any bypass at the bottom, but I personally never recommend that to anyone. I have customers set up rules to bypass whatever they need to bypass and then any any inspect at the bottom (which is default anyway). I find when you have it that way, I never see any traffic or acceleration issues at all.

Just my 2 cents.

Wolfgang · ‎2023-02-08

One more to be aware…. Don‘t set the log level of your last bypass rule to „detailed“ or „extended log“. This will mitigate the acceleration.

the_rock · ‎2023-02-08

correct

Timothy_Hall · ‎2023-02-08

Can you please clarify your statement @Wolfgang? It doesn't seem to be possible to set a Track log value of Detailed or Extended in the HTTPS Inspection Policy. Were you referring to one of the other Access Control layers?

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com

Wolfgang · ‎2023-02-08

@Timothy_Hall yes, you're right my mistake not to explain further. I'm referring to the access control layer, especially the URLF/APPCL layer. With a lot of HTTPS inspected or bypassed traffic and a match of these traffic in an URLF/APPCL layer the log setting will be significant for the performance. I think you mentioned this in your book.

@andy_currigan started here with HTTPS inspection performance questions regarding the rule order but I think you have to observe the other performance indicators too.

Gojira · ‎2023-03-01

Any luck here?
I still couldn't find the culprit.

Are you a member of CheckMates?

https inspection performance issue - F2F traffic 82%..