topic Re: https inspection performance issue - F2F traffic 82%.. in Firewall and Security Management

https inspection performance issue - F2F traffic 82%..

andy_currigan — Mon, 06 Feb 2023 15:29:38 GMT

we have 2x open server in cluster XL r81 with a 4x cpu license

running https://openspeedtest.com/ (run on https) we notice an important drop in terms of performance, instead of 450-500 mbps that we get from a standard speedtest liko ookla that run on port 8080 the performance drops to max 150 mbps

we investigate the https inspection module but we do not understand the following behaviour.

1) if we create a rule in first position that bypass the entire https inspection for a specific host openspeedtest run at 450 mbps but if we insert the same host in a rule in position 5 that bypass the https inspection if you're member of a group of hosts the same host run at 150 mbps.

To note that in both case the url https://openspeedtest.com/ is bypassed due his categorization..

how is possible such a performance drop based on the https rule position?

why there's such a drop considering that this service is not even inspected?

we did some debug and we notice that lot's of traffic is goes through F2F below some usefull outputs.

any suggestions? tx

[Expert@checkpoint-1:0]# fwaccel stats -s
Accelerated conns/Total conns : 135/1570 (8%)
Accelerated pkts/Total pkts : 13101094681/73808428002 (17%)
F2Fed pkts/Total pkts : 60707333321/73808428002 (82%)
F2V pkts/Total pkts : 65943865/73808428002 (0%)
CPASXL pkts/Total pkts : 16454023/73808428002 (0%)
PSLXL pkts/Total pkts : 12372846626/73808428002 (16%)
CPAS pipeline pkts/Total pkts : 0/73808428002 (0%)
PSL pipeline pkts/Total pkts : 0/73808428002 (0%)
CPAS inline pkts/Total pkts : 0/73808428002 (0%)
PSL inline pkts/Total pkts : 0/73808428002 (0%)
QOS inbound pkts/Total pkts : 0/73808428002 (0%)
QOS outbound pkts/Total pkts : 0/73808428002 (0%)
Corrected pkts/Total pkts : 0/73808428002 (0%)

Accept Templates : enabled
Drop Templates : enabled
NAT Templates : enabled
[Expert@checkpoint-1:0]#

[Expert@checkpoint-1:0]# fwaccel stats
Name Value Name Value
---------------------------- ------------ ---------------------------- ------------

Accelerated Path
--------------------------------------------------------------------------------------
accel packets 13105476237 accel bytes 9576220636092
outbound packets 13306779203 outbound bytes 9868235245852
conns created 65305430 conns deleted 65304144
C total conns 1286 C TCP conns 478
C non TCP conns 808 nat conns 31963228
dropped packets 2907232 dropped bytes 629465065
fragments received 1831431 fragments transmit 1126
fragments dropped 0 fragments expired 111439
IP options stripped 374709 IP options restored 115724
IP options dropped 0 corrs created 0
corrs deleted 0 C corrections 0
corrected packets 0 corrected bytes 0

Accelerated VPN Path
--------------------------------------------------------------------------------------
C crypt conns 115 enc bytes 201667504
dec bytes 682230912 ESP enc pkts 528421
ESP enc err 0 ESP dec pkts 708080
ESP dec err 0 ESP other err 0
espudp enc pkts 0 espudp enc err 0
espudp dec pkts 0 espudp dec err 0
espudp other err 0

Medium Streaming Path
--------------------------------------------------------------------------------------
CPASXL packets 16454121 PSLXL packets 12377039895
CPASXL async packets 16454023 PSLXL async packets 12376375499
CPASXL bytes 16508578013 PSLXL bytes 8937294264245
C CPASXL conns 3 C PSLXL conns 1157
CPASXL conns created 50783 PSLXL conns created 64697868
PXL FF conns 0 PXL FF packets 29416
PXL FF bytes 23581461 PXL FF acks 12056
PXL no conn drops 0

Pipeline Streaming Path
--------------------------------------------------------------------------------------
PSL Pipeline packets 0 PSL Pipeline bytes 0
CPAS Pipeline packets 0 CPAS Pipeline bytes 0

Inline Streaming Path
--------------------------------------------------------------------------------------
PSL Inline packets 0 PSL Inline bytes 0
CPAS Inline packets 0 CPAS Inline bytes 0

Buffer Path
--------------------------------------------------------------------------------------
Buffer path buffers 0 Buffer path bytes 0

TLS PARSER
--------------------------------------------------------------------------------------
RECORD INFO 0

TLS DECRYPT
--------------------------------------------------------------------------------------
TLS INSPECTION 0 TLS HANDSHAKE 0
TLS RECORD LAYER 0 TLS CRYPTO 0

HTTP DISP
--------------------------------------------------------------------------------------
ACTIVATE WS MAIN 0 EXEC NO HTTP CMI CONTEXT 0

WS LITE
--------------------------------------------------------------------------------------
WS TX COMPLETED 0 WS FORWARD TO MAIN 0
WS NOTIFY TIMEOUT 0 WS HANDLE EVENT 0
WS CHUNKED ERROR 0 WS GZIP EVENT 0
WS ADD MAC HEADER 0 WS IS STICKY ACTIVE 0
WS TIER1 JOB ERROR 0 WS TIER1 HAS MATCHES 0
CML MATCHES 0 TOTAL UPLOADED JOBS 0
TOTAL JOBS 0

ADVP
--------------------------------------------------------------------------------------
ADVP FORW TO MAIN 0 ADVP HOLD TIMEOUT 0

QoS Paths
--------------------------------------------------------------------------------------
QoS General Information:
------------------------
Total QoS Conns 0 QoS Classify Conns 0
QoS Classify flow 0 Reclassify QoS policy 0

FireWall QoS Path:
------------------
Enqueued IN packets 0 Enqueued OUT packets 0
Dequeued IN packets 0 Dequeued OUT packets 0
Enqueued IN bytes 0 Enqueued OUT bytes 0
Dequeued IN bytes 0 Dequeued OUT bytes 0

Accelerated QoS Path:
---------------------
Enqueued IN packets 0 Enqueued OUT packets 0
Dequeued IN packets 0 Dequeued OUT packets 0
Enqueued IN bytes 0 Enqueued OUT bytes 0
Dequeued IN bytes 0 Dequeued OUT bytes 0

Firewall Path
--------------------------------------------------------------------------------------
F2F packets 60736129970 F2F bytes 50320444505511
TCP violations 16 F2V conn match pkts 703989
F2V packets 65971694 F2V bytes 5570164544

GTP
--------------------------------------------------------------------------------------
gtp tunnels created 0 gtp tunnels 0
gtp accel pkts 0 gtp f2f pkts 0
gtp spoofed pkts 0 gtp in gtp pkts 0
gtp signaling pkts 0 gtp tcpopt pkts 0
gtp apn err pkts 0

General
--------------------------------------------------------------------------------------
memory used 103915120 C tcp handshake conns 14
C tcp established conns 454 C tcp closed conns 10
C tcp pxl handshake conns 14 C tcp pxl established conns 351
C tcp pxl closed conns 10 DNS DoR stats 291

(*) Statistics marked with C refer to current value, others refer to total value

[Expert@checkpoint-1:0]#

ragione del non accelerazione

[Expert@checkpoint-1:0]# fwaccel stats -p
F2F packets:
--------------
Violation Packets Violation Packets
-------------------- --------------- -------------------- ---------------
Pkt has IP options 374766 ICMP miss conn 249518305
TCP-SYN miss conn 215843802 TCP-other miss conn 29156577762
UDP miss conn 920603464 Other miss conn 5798272
VPN returned F2F 128716 Uni-directional viol 0
Possible spoof viol 0 TCP state viol 547
SCTP state affecting 0 Out if not def/accl 0
Bridge src=dst 0 Routing decision err 0
Sanity checks failed 0 Fwd to non-pivot 0
Broadcast/multicast 0 Cluster message 109434977
Cluster forward 635 Chain forwarding 0
F2V conn match pkts 705245 General reason 0
Route changes 0 VPN multicast traffic 0
GTP non-accelerated 0 Unresolved nexthop 38438
[Expert@checkpoint-1:0]# fwaccel stats -t
F2Fed bytes/Total bytes : 50329254032099/59906647972752 (84%)
F2V bytes/Total bytes : 5571144798/59906647972752 (0%)
Medium path bytes/Total bytes : 8954924374718/59906647972752 (14%)
Pipeline path bytes/Total bytes : 0/59906647972752 (0%)
Inline path bytes/Total bytes : 0/59906647972752 (0%)
Buffer path bytes/Total inline bytes: 0/0 (0%)

[Expert@checkpoint-1:0]# fw ctl affinity -l -r
CPU 0:
CPU 1: fw_1 (active)
mpdaemon fwd rad lpd rtmd wsdnsd in.asessiond core_uploader cprid usrchkd vpnd in.acapd pepd pdpd cprid cpd
CPU 2:
CPU 3:
CPU 4:
CPU 5:
CPU 6:
CPU 7:
CPU 8:
CPU 9: fw_0 (active)
mpdaemon fwd rad lpd rtmd wsdnsd in.asessiond core_uploader cprid usrchkd vpnd in.acapd pepd pdpd cprid cpd
CPU 10:
CPU 11:
CPU 12:
CPU 13:
CPU 14:
CPU 15:
All:
The current license permits the use of CPUs 0, 1, 8, 9 only.
Interface eth4: has multi queue enabled
Interface eth5: has multi queue enabled
Interface eth0: has multi queue enabled
Interface eth6: has multi queue enabled
Interface eth3: has multi queue enabled
[Expert@checkpoint-1:0]# fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 9 | 5940 | 11921
1 | Yes | 1 | 6379 | 13536

[Expert@checkpoint-1:0]# enabled_blades
fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot mon

Re: https inspection performance issue - F2F traffic 82%..

PhoneBoy — Mon, 06 Feb 2023 22:30:09 GMT

You said R81, what JHF level?
The fact you see this issue when you put the host in Rule 5 (versus as a separate rule up top) suggests that the rules before Rule 5 are potentially problematic.
Please provide the precise rules used in your HTTPS Inspection policy (specifically up to and including Rule 5)..

Note that, in general, bypass rules should be listed first.

Re: https inspection performance issue - F2F traffic 82%..

Chris_Atkinson — Mon, 06 Feb 2023 23:41:08 GMT

Recommend reviewing HTTPS inspection policy structure per the recommendations here & here to start.

Refer also:

sk98871 : HTTP-based speed test shows strong degradation in download speed when Anti-Virus is set to inspect all HTTP and HTTPS traffic

sk163595 : HTTPS Inspection bypass list object

Re: https inspection performance issue - F2F traffic 82%..

Timothy_Hall — Tue, 07 Feb 2023 02:19:12 GMT

As Chris said you must follow the rules order precisely as specified in those two links. The reason it is so important to have the rules in the correct order is due to the two-pass matching on the HTTPS Inspection policy. The first pass happens with the first packet of a new connection based on just IP addresses and ports. If a matching bypass rule can be found with this limited information (no domains/sites/categories allowed before any matching bypass rule here) active streaming is avoided and the connection can be passively streamed in the medium path or even fully accelerated by SecureXL.

If a matching Bypass cannot be found in the first pass, active streaming must be invoked to determine domain/site/category in the next few packets of the connection which incurs significant overhead, then the second pass occurs.

Are you sure you didn't run that fwaccel stats -s command on the standby member of your cluster? High F2F is expected there. HTTPS Inspection alone should not cause high F2F on the active member, something else is causing that, probably IPS. Try the fw tab -f -u -z -t connections command to see exclusively what connections are F2F and what they have in common. If that doesn't work try fw_streaming path slow (not sure on syntax there, typing all this on my phone). It looks like the main F2F violation causing the high F2F is "TCP other miss conn". Too generic so a debug must be run to find reason for F2F.

Everything I just posted is described and executed in labs (including the F2F debug) in my new live R81.20 Gateway Performance Optimization class.

Re: https inspection performance issue - F2F traffic 82%..

andy_currigan — Wed, 08 Feb 2023 11:47:13 GMT

r81 take 77

the problem seems not related to the https rules order. the rule set is very simple with rules 1 to 5 that bypass several hosts by source, by destination ip or by url then we have an ispection rule for some dangerous categories and at the end the bypass for any as suggested in the following link

Solved: Re: HTTPS Inspection Setup - Check Point CheckMates

the problem cause seems related to the high percentage of traffic that pass through the slow path.

Re: https inspection performance issue - F2F traffic 82%..

andy_currigan — Wed, 08 Feb 2023 11:48:29 GMT

Solved: Re: HTTPS Inspection Setup - Check Point CheckMates

the problem cause seems related to the high percentage of traffic that pass through the slow path.

Re: https inspection performance issue - F2F traffic 82%..

Chris_Atkinson — Wed, 08 Feb 2023 12:26:02 GMT

sk32578 describes common scenarios causing F2F traffic processing e.g. PPPoE connections and the use of certain object types / IPS protections (sk105079) etc.

As Tim said high F2F stats are also expected on a standby member of a cluster.

SQLNET2 traffic may also be a factor per sk179919.

Re: https inspection performance issue - F2F traffic 82%..

Timothy_Hall — Wed, 08 Feb 2023 12:57:40 GMT

As I mentioned earlier HTTPS Inspection by itself does not typically cause high F2F. It is almost certainly due to the configuration of your IPS blade. Try the following technique which is mentioned in my Gateway Performance Optimization Course during your firewall's busiest period if possible, keeping in mind doing so may expose your organization to attacks, use at your own risk!

fwaccel stats -s (note F2F percentage)

ips off -n

fwaccel stats -r

(wait 120 seconds)

fwaccel stats -s (has F2F dropped substantially? If so IPS is to blame)

ips on

Re: https inspection performance issue - F2F traffic 82%..

Machine_Head — Wed, 08 Feb 2023 16:35:33 GMT

Not trying to hijack your post but to add that I am having a similar issue and might be related.

Env:
Maestro R81.10 JHF 78

Mostly all blades enabled AND HTTPSi

Issue:

Usually F2F 95%

Have unloaded amw policy and checked stats and it certainly decreases, but F2F still around 65%, too much.

Main violation is TCP-other miss conn

In cpview, offload decision says SPII Active

Run debug:

fwaccel dbg + offload

fwaccel dbg + pkt + f2f

And I see:

;cphwd_chain_to_vm_connkey: obtained vm_connkey <dir 1, x:58180 -> x:443 IPP 6>;
@;235926526; 8Feb2023 15:47:37.404890;[vs_0];[tid_2];[fw4_2];cphwd_chain_to_vm_connkey: obtained vm_connkey <dir 1, x:58180 -> 52.97.211.226:443 IPP 6>;
@;235926526; 8Feb2023 15:47:37.404895;[vs_0];[tid_2];[fw4_2];cpxl_chain_handler: entered (conn <dir 0, 10.x7:58180 -> 52.x6:443 IPP 6>);
@;235926526; 8Feb2023 15:47:37.404897;[vs_0];[tid_2];[fw4_2];cphwd_is_conn_already_offloaded: conn_already_offloaded=0, reoffload_conn=0, created_from_template=0, policy_id=354827077, up_policy_id=1675859405;
@;235926526; 8Feb2023 15:47:37.404898;[vs_0];[tid_2];[fw4_2];cphwd_chain_to_vm_connkey: obtained vm_connkey <dir 1, 1x:58180 -> 52.x:443 IPP 6>;
@;235926526; 8Feb2023 15:47:37.404900;[vs_0];[tid_2];[fw4_2];cphwd_is_non_accelerated_conn: entered...orig_conn=<dir 0, 1x:58180 -> 52.x:443 IPP 6>, vm_conn=<dir 1, 10.x7:58180 -> 52.9x:443 IPP 6>;
@;235926526; 8Feb2023 15:47:37.404903;[vs_0];[tid_2];[fw4_2];cphwd_conn_should_accelerate_by_features: spii is active -> not offloading;
@;235926526; 8Feb2023 15:47:37.404904;[vs_0];[tid_2];[fw4_2];cphwd_handle_conn_acceleration: cphwd_conn_should_accelerate_by_features - not accelerating;
@;235926526; 8Feb2023 15:47:37.404912;[vs_0];[tid_2];[fw4_2];cpxl_chain_handler: connection <dir 0, x58180 -> 5x6:443 IPP 6> will not be accelerated;

and then for return traffic:

;pkt_handle_no_match: connection <52.x,443,10.x,55119,6> not found -> forwarding to firewall;

Flags for slow connections if it means anything to somebody

Path: Slow, InZone: INTERNAL_ZONE, OutZone: EXTERNAL_ZONE, Flags: 0x4b01
C2S side:
TCP State: PSL_TCP_ESTABLISHED, Number of segments: 0, Hold: 0, Side flags: 0x2c4
S2C side:
TCP State: PSL_TCP_ESTABLISHED, Number of segments: 0, Hold: 0, Side flags: 0x2c4
Application info:
MUX_PASSIVE, Flags: 0x13

Connection matches an HTTPSi bypass rule as well.

Currently with TAC but still early stages.. they are still beating around the bush

Re: https inspection performance issue - F2F traffic 82%..

Timothy_Hall — Wed, 08 Feb 2023 16:35:34 GMT

SPII = IPS.

Amw unload does not stop IPS. After ips off -n and stats reset you may need to wait longer if you have long running connections as they will continue F2F until they end.

Re: https inspection performance issue - F2F traffic 82%..

Machine_Head — Wed, 08 Feb 2023 16:45:25 GMT

Any debug or way to find out the protection causing it?

Re: https inspection performance issue - F2F traffic 82%..

CheckPointerXL — Wed, 08 Feb 2023 17:01:00 GMT

Hello Tim,

i've got list of F2F connections by executing fw tab -t connections -z (altough seems to display not only F2F because of no value under Not Offloaded Reason); how can i easily identify most impacting connections?

PS: fw_streaming path slow seems to be very fantasious, no way to guess correct syntax 😄

and

fw tab -f -u -z -t connections
Using cptfmt
Formatting table's data - this might take a while...

-z option must be used with connections table

Re: https inspection performance issue - F2F traffic 82%..

Timothy_Hall — Wed, 08 Feb 2023 16:57:33 GMT

Yes, run hcp's "secret" Threat Prevention reports:

hcp --enable-product "Threat Prevention"

hcp -r "Threat Prevention"

You may need to update to the latest version of hcp and/or these reports may not be supported on R81.10. I am currently attending CPX Denver for Partners in person and will post the relevant pages from my Gateway Performance Optimization course later today.

Re: https inspection performance issue - F2F traffic 82%..

the_rock — Wed, 08 Feb 2023 17:14:48 GMT

I read all the comments and responses given and here is all I can say. I know lots of people advise when it comes to https inspection to use rule any any bypass at the bottom, but I personally never recommend that to anyone. I have customers set up rules to bypass whatever they need to bypass and then any any inspect at the bottom (which is default anyway). I find when you have it that way, I never see any traffic or acceleration issues at all.

Just my 2 cents.

Re: https inspection performance issue - F2F traffic 82%..

Wolfgang — Wed, 08 Feb 2023 17:50:07 GMT

One more to be aware…. Don‘t set the log level of your last bypass rule to „detailed“ or „extended log“. This will mitigate the acceleration.

Re: https inspection performance issue - F2F traffic 82%..

Timothy_Hall — Wed, 08 Feb 2023 18:16:35 GMT

To find the most impacting connections run the secret hcp Threat Prevention reports I mentioned elsewhere in this thread, as it identifies top elephant flows subject to TP inspection.

As far as fw_streaming it looks like the "path slow" argument only works in R81.20+.

Re: https inspection performance issue - F2F traffic 82%..

Timothy_Hall — Thu, 09 Feb 2023 01:36:26 GMT

Here are the pages from my Gateway Performance Optimization course covering the secret hcp TP reports with screenshots, as you can see there is a vast wealth of useful information.

Re: https inspection performance issue - F2F traffic 82%..

the_rock — Thu, 09 Feb 2023 01:41:19 GMT

correct

Re: https inspection performance issue - F2F traffic 82%..

Timothy_Hall — Thu, 09 Feb 2023 01:59:37 GMT

Can you please clarify your statement @Wolfgang? It doesn't seem to be possible to set a Track log value of Detailed or Extended in the HTTPS Inspection Policy. Were you referring to one of the other Access Control layers?

Re: https inspection performance issue - F2F traffic 82%..

Wolfgang — Thu, 09 Feb 2023 07:33:30 GMT

@Timothy_Hall yes, you're right my mistake not to explain further. I'm referring to the access control layer, especially the URLF/APPCL layer. With a lot of HTTPS inspected or bypassed traffic and a match of these traffic in an URLF/APPCL layer the log setting will be significant for the performance. I think you mentioned this in your book.

@andy_currigan started here with HTTPS inspection performance questions regarding the rule order but I think you have to observe the other performance indicators too.