HTTPS Inspection significantly slows down loading of images

[konecnyl]

Hi,

on our firewall (Check Point R81.20 - Build 043), we are observing significant slowdowns when loading images while using map services – especially when panning or zooming maps on portals such as mapy.cz, Google Maps, or Street View.
The issue affects only images (tiles), while other parts of the pages load smoothly.

For selected services (e.g. mapserver.mapy.cz, panorama-mapserver.mapy.cz), we created Custom Applications/Sites and added them to bypass rules for HTTPS Inspection, which helped. However, on other portals where a large number of image requests occur, the slowdowns still persist.

Context:

• Firewall: R81.20 - Build 043, KSFW, 16 cores, 14 fw_worker instances

• QUIC is blocked

• HTTPS Inspection is enabled with categorization, 0% accelerated connections

• CPASXL traffic is around 40% of total traffic

• Latency increases, especially with a larger number of image requests

• Browsers: Chrome/Edge/Firefox (latest versions), OS: Windows 10/11

• MS Defender and Windows policies are up-to-date

• Perf and Spike Detective show load on fw_worker threads (e.g. kiss_thin_nfa_exec_one_buf_parallel_xlate, cmi_execute_ex)

Questions:

• Do you have any recommendations on how to effectively deal with slowdowns during inspection of large numbers of image requests?

• Is there a way to bypass only image requests (e.g. by MIME type, file path, or pattern), but inspect all other traffic?

• Has anyone seen improvements by switching to USFW mode, if the issue was related only to TLS inspection of map tiles?

• Could recent Windows or browser updates have impacted TLS handshake behavior and caused additional inspection load?

Thanks for any insight – we welcome even small real-world experiences.

[PhoneBoy]

It is not technically feasible to bypass HTTPS Inspection for a specific resource (type) on a page as the decision to bypass or inspect must be made before the encryption is negotiated.

The only reason I can think of for images specifically to be slower is because they are being processed by a Threat Prevention blade.
What Threat Prevention blades are active on this gateway and do you see any logs that might indicate this?

[the_rock]

Maybe make sure what I attached is set to background, it can help.

Andy

[Chris_Atkinson]

Thanks for providing the major version used, what JHF/Jumbo take & enabled_blades?

[Lesley]

indeed need to know jumbo to see open bugs.

+ have you seen optional and recommended bypass in this SK? https://support.checkpoint.com/results/sk/sk163595

if complains are with google worth to bypass that is listed here. 

Do you have any RAD errors? (a few is OK, loads is worth checking)

https://support.checkpoint.com/results/sk/sk182859

[konecnyl]

Thanks for your reply! Here are the details:

• Version: R81.20

• Jumbo Hotfix: Take 92

• Enabled Blades: Firewall, Application Control, URL Filtering, HTTPS Inspection, Identity Awareness, Threat Emulation, Anti-Virus, Anti-Bot, IPS

[Chris_Atkinson]

Take 99 (bundling fixes from previous JHFs) includes some AppC fixes including for gzip encoded flows, possibly relevant.

[the_rock]

Definitely good idea Chris.

I do see below in the notes as well.

Andy

PRJ-52612, PRHF-31058

SSL Inspection

NEW: This Take introduces a fail-open mechanism for HTTPS Inspection with Hardware Security Module (HSM) integration. If the HSM becomes unavailable, TLS connections now automatically bypass HTTPS Inspection, ensuring continuous network connectivity.

[the_rock]

Let me take a crack at the questions...

Andy

• Do you have any recommendations on how to effectively deal with slowdowns during inspection of large numbers of image requests?

I would check background vs hold mode settings I attached in my screenshots in the previous response

• Is there a way to bypass only image requests (e.g. by MIME type, file path, or pattern), but inspect all other traffic?

Not that Im aware of, unless you know the EXACT fqdn

• Has anyone seen improvements by switching to USFW mode, if the issue was related only to TLS inspection of map tiles?

Personally, I never had an issue regardless of the mode I was using

• Could recent Windows or browser updates have impacted TLS handshake behavior and caused additional inspection load?

Definitely not, as I had not had any issues in my lab due to those updates

[Timothy_Hall · Accepted solution]

Do you have Policy-Based Routing enabled?  There is a known problem with PBR and Active Streaming: sk183194: Slow web browsing to the Internet and download of files is stuck

Also make sure that OCSP and CRL Retrieval are working: Unreached OCSP - causes serious lag

If you have persistent high latency for Internet traffic, there was an issue with Active Streaming where it would not allow the TCP window to scale up far enough to counteract the high latency, but that should have been fixed awhile ago: HTTPS Inspection Limiting TCP receive window to 262144 bytes and limiting throughput of tcp stream

Also be very sure you are NOT using "Any" in the Destination or Services fields of any HTTPS Inspection rules, as this will drag huge amounts of traffic into Active Streaming that should not be there.

Finally you may need to do some tuning of Active Streaming by adjusting the variable cpas_max_burst among others as mentioned here: https://community.checkpoint.com/t5/Security-Gateways/81-20-Performance-CPU-issue/m-p/214709#M40990

[konecnyl]

So far, we've identified the following issues in the logs:

RAD request exceeded maximum handling time, check /opt/CPsuite-R81.20/fw1/log/rad_events/Errors/flow_61468_20244022 for more details

[rad_http_response_handler.cpp:27] CRadHttpResponseHandler::goToNextTask: [INFO] enter to ...
[rad_response_task.cpp:19] CRadResponseTask::CRadResponseTask: [INFO] enter to ...
[rad_decrypted_response_task.cpp:24] CRadDecryptedResponseTask::CRadDecryptedResponseTask: [INFO] enter to ...

Unreached OCSP (HTTPS Inspection blade):
Failed to fetch OCSP. Make sure the security gateway has an outgoing HTTP access, and that the proxy and DNS servers are well configured.
Refer to sk159872 for more details.
Certificate DN: 'CN=cas.avalon.perfdrive.com'
Requested Server Name: cas.avalon.perfdrive.com

[PhoneBoy]

The inability to fetch OCSP from the gateway would definitely cause the slowness issues.
Specifically, these connections are made from the gateway to verify the certificate on the remote end.
Ensure the gateway has outgoing HTTP access and proxy/DNS settings are correct.

[konecnyl]

In the end, it turned out that applying the hotfix to Take 99 helped – the response time is now instant.
We have PBR enabled, so we followed sk183194: Slow web browsing to the Internet and download of files is stuck.
We also removed the newly created bypass rules. All related processes, such as fw_worker and rad (which was previously at 290% CPU), have significantly dropped in the top output.

So far, I haven't had the chance to test it thoroughly, mainly due to lower employee activity today, but it looks like the overall CPU load on the gateway has also improved.

Because of the lower usage today, I will do full testing again on Monday – but I already dare to say that the issue is resolved.

Thank you for your valuable advice – it helped a lot.
And I apologize for the delayed replies and maybe some nervousness.

Lukas

[konecnyl]

After further investigation, we found that the issue is not limited to image loading – it's a general slowness in browsing websites and internet access. When we apply a full HTTPS Inspection bypass for the affected user, the response time becomes instant and the issue disappears.

[the_rock]

Did you check settings from the screenshots I sent initially? I really believe if those are wrong, it could cause such issue.

Andy

[konecnyl]

Yes, I checked the settings from your screenshots.

The first option is configured exactly as in your screenshot.

For the second option, I changed it from Custom to Background, but unfortunately it did not improve the situation.

Lukas

[the_rock]

Just saw your last response...did you check with TAC if it indeed could be RAD related? If so, maybe check out below discussion.

Andy

https://community.checkpoint.com/t5/Security-Gateways/URL-filtering-blade-RAD-process-causing-high-C...

[konecnyl]

In the end, it turned out that applying the hotfix to Take 99 helped – the response time is now instant.
We have PBR enabled, so we followed sk183194: Slow web browsing to the Internet and download of files is stuck.
We also removed the newly created bypass rules. All related processes, such as fw_worker and rad (which was previously at 290% CPU), have significantly dropped in the top output.

So far, I haven't had the chance to test it thoroughly, mainly due to lower employee activity today, but it looks like the overall CPU load on the gateway has also improved.

Because of the lower usage today, I will do full testing again on Monday – but I already dare to say that the issue is resolved.

Thank you for your valuable advice – it helped a lot.
And I apologize for the delayed replies and maybe some nervousness.

Lukas

[the_rock]

Awesome job!