topic Re: HTTPS Inspection significantly slows down loading of images in Firewall and Security Management

HTTPS Inspection significantly slows down loading of images

konecnyl — Wed, 07 May 2025 20:13:44 GMT

Hi,

on our firewall (Check Point R81.20 - Build 043), we are observing significant slowdowns when loading images while using map services – especially when panning or zooming maps on portals such as mapy.cz, Google Maps, or Street View.
The issue affects only images (tiles), while other parts of the pages load smoothly.

For selected services (e.g. mapserver.mapy.cz, panorama-mapserver.mapy.cz), we created Custom Applications/Sites and added them to bypass rules for HTTPS Inspection, which helped. However, on other portals where a large number of image requests occur, the slowdowns still persist.

Context:

Firewall: R81.20 - Build 043, KSFW, 16 cores, 14 fw_worker instances
QUIC is blocked
HTTPS Inspection is enabled with categorization, 0% accelerated connections
CPASXL traffic is around 40% of total traffic
Latency increases, especially with a larger number of image requests
Browsers: Chrome/Edge/Firefox (latest versions), OS: Windows 10/11
MS Defender and Windows policies are up-to-date
Perf and Spike Detective show load on fw_worker threads (e.g. kiss_thin_nfa_exec_one_buf_parallel_xlate, cmi_execute_ex)

Questions:

Do you have any recommendations on how to effectively deal with slowdowns during inspection of large numbers of image requests?
Is there a way to bypass only image requests (e.g. by MIME type, file path, or pattern), but inspect all other traffic?
Has anyone seen improvements by switching to USFW mode, if the issue was related only to TLS inspection of map tiles?
Could recent Windows or browser updates have impacted TLS handshake behavior and caused additional inspection load?

Thanks for any insight – we welcome even small real-world experiences.

Re: HTTPS Inspection significantly slows down loading of images

PhoneBoy — Wed, 07 May 2025 21:01:38 GMT

It is not technically feasible to bypass HTTPS Inspection for a specific resource (type) on a page as the decision to bypass or inspect must be made before the encryption is negotiated.

The only reason I can think of for images specifically to be slower is because they are being processed by a Threat Prevention blade.
What Threat Prevention blades are active on this gateway and do you see any logs that might indicate this?

Re: HTTPS Inspection significantly slows down loading of images

the_rock — Wed, 07 May 2025 21:46:27 GMT

Maybe make sure what I attached is set to background, it can help.

Andy

Re: HTTPS Inspection significantly slows down loading of images

Chris_Atkinson — Wed, 07 May 2025 23:37:21 GMT

Thanks for providing the major version used, what JHF/Jumbo take & enabled_blades?

Re: HTTPS Inspection significantly slows down loading of images

the_rock — Thu, 08 May 2025 01:09:59 GMT

Let me take a crack at the questions...

Andy

Do you have any recommendations on how to effectively deal with slowdowns during inspection of large numbers of image requests?

I would check background vs hold mode settings I attached in my screenshots in the previous response

Is there a way to bypass only image requests (e.g. by MIME type, file path, or pattern), but inspect all other traffic?

Not that Im aware of, unless you know the EXACT fqdn

Has anyone seen improvements by switching to USFW mode, if the issue was related only to TLS inspection of map tiles?

Personally, I never had an issue regardless of the mode I was using

Could recent Windows or browser updates have impacted TLS handshake behavior and caused additional inspection load?

Definitely not, as I had not had any issues in my lab due to those updates

Re: HTTPS Inspection significantly slows down loading of images

Timothy_Hall — Thu, 08 May 2025 12:36:37 GMT

Do you have Policy-Based Routing enabled? There is a known problem with PBR and Active Streaming: sk183194: Slow web browsing to the Internet and download of files is stuck

Also make sure that OCSP and CRL Retrieval are working: Unreached OCSP - causes serious lag

If you have persistent high latency for Internet traffic, there was an issue with Active Streaming where it would not allow the TCP window to scale up far enough to counteract the high latency, but that should have been fixed awhile ago: HTTPS Inspection Limiting TCP receive window to 262144 bytes and limiting throughput of tcp stream

Also be very sure you are NOT using "Any" in the Destination or Services fields of any HTTPS Inspection rules, as this will drag huge amounts of traffic into Active Streaming that should not be there.

Finally you may need to do some tuning of Active Streaming by adjusting the variable cpas_max_burst among others as mentioned here: https://community.checkpoint.com/t5/Security-Gateways/81-20-Performance-CPU-issue/m-p/214709#M40990

Re: HTTPS Inspection significantly slows down loading of images

Lesley — Thu, 08 May 2025 20:38:12 GMT

indeed need to know jumbo to see open bugs.

+ have you seen optional and recommended bypass in this SK? https://support.checkpoint.com/results/sk/sk163595

if complains are with google worth to bypass that is listed here.

Do you have any RAD errors? (a few is OK, loads is worth checking)

https://support.checkpoint.com/results/sk/sk182859

Re: HTTPS Inspection significantly slows down loading of images

konecnyl — Fri, 09 May 2025 06:55:51 GMT

Thanks for your reply! Here are the details:

Version: R81.20
Jumbo Hotfix: Take 92
Enabled Blades: Firewall, Application Control, URL Filtering, HTTPS Inspection, Identity Awareness, Threat Emulation, Anti-Virus, Anti-Bot, IPS

Re: HTTPS Inspection significantly slows down loading of images

konecnyl — Fri, 09 May 2025 10:52:01 GMT

After further investigation, we found that the issue is not limited to image loading – it's a general slowness in browsing websites and internet access. When we apply a full HTTPS Inspection bypass for the affected user, the response time becomes instant and the issue disappears.

Re: HTTPS Inspection significantly slows down loading of images

the_rock — Fri, 09 May 2025 10:55:14 GMT

Did you check settings from the screenshots I sent initially? I really believe if those are wrong, it could cause such issue.

Andy

Re: HTTPS Inspection significantly slows down loading of images

konecnyl — Fri, 09 May 2025 12:48:29 GMT

Yes, I checked the settings from your screenshots.

The first option is configured exactly as in your screenshot.

For the second option, I changed it from Custom to Background, but unfortunately it did not improve the situation.

Lukas

Re: HTTPS Inspection significantly slows down loading of images

konecnyl — Fri, 09 May 2025 12:55:09 GMT

So far, we've identified the following issues in the logs:

RAD request exceeded maximum handling time, check /opt/CPsuite-R81.20/fw1/log/rad_events/Errors/flow_61468_20244022 for more details

[rad_http_response_handler.cpp:27] CRadHttpResponseHandler::goToNextTask: [INFO] enter to ...
[rad_response_task.cpp:19] CRadResponseTask::CRadResponseTask: [INFO] enter to ...
[rad_decrypted_response_task.cpp:24] CRadDecryptedResponseTask::CRadDecryptedResponseTask: [INFO] enter to ...

Unreached OCSP (HTTPS Inspection blade):
Failed to fetch OCSP. Make sure the security gateway has an outgoing HTTP access, and that the proxy and DNS servers are well configured.
Refer to sk159872 for more details.
Certificate DN: 'CN=cas.avalon.perfdrive.com'
Requested Server Name: cas.avalon.perfdrive.com

Re: HTTPS Inspection significantly slows down loading of images

the_rock — Fri, 09 May 2025 12:59:51 GMT

Just saw your last response...did you check with TAC if it indeed could be RAD related? If so, maybe check out below discussion.

Andy

https://community.checkpoint.com/t5/Security-Gateways/URL-filtering-blade-RAD-process-causing-high-CPU-tip/m-p/246430#M48026

Re: HTTPS Inspection significantly slows down loading of images

Chris_Atkinson — Fri, 09 May 2025 13:02:43 GMT

Take 99 (bundling fixes from previous JHFs) includes some AppC fixes including for gzip encoded flows, possibly relevant.

Re: HTTPS Inspection significantly slows down loading of images

the_rock — Fri, 09 May 2025 13:25:55 GMT

Definitely good idea Chris.

I do see below in the notes as well.

Andy

PRJ-52612,
PRHF-31058

SSL Inspection

NEW: This Take introduces a fail-open mechanism for HTTPS Inspection with Hardware Security Module (HSM) integration. If the HSM becomes unavailable, TLS connections now automatically bypass HTTPS Inspection, ensuring continuous network connectivity.

Re: HTTPS Inspection significantly slows down loading of images

PhoneBoy — Fri, 09 May 2025 16:12:49 GMT

The inability to fetch OCSP from the gateway would definitely cause the slowness issues.
Specifically, these connections are made from the gateway to verify the certificate on the remote end.
Ensure the gateway has outgoing HTTP access and proxy/DNS settings are correct.

Re: HTTPS Inspection significantly slows down loading of images

konecnyl — Fri, 09 May 2025 18:27:11 GMT

In the end, it turned out that applying the hotfix to Take 99 helped – the response time is now instant.
We have PBR enabled, so we followed sk183194: Slow web browsing to the Internet and download of files is stuck.
We also removed the newly created bypass rules. All related processes, such as fw_worker and rad (which was previously at 290% CPU), have significantly dropped in the top output.

So far, I haven't had the chance to test it thoroughly, mainly due to lower employee activity today, but it looks like the overall CPU load on the gateway has also improved.

Because of the lower usage today, I will do full testing again on Monday – but I already dare to say that the issue is resolved.

Thank you for your valuable advice – it helped a lot.
And I apologize for the delayed replies and maybe some nervousness.

Lukas

Re: HTTPS Inspection significantly slows down loading of images

konecnyl — Fri, 09 May 2025 18:27:42 GMT

So far, I haven't had the chance to test it thoroughly, mainly due to lower employee activity today, but it looks like the overall CPU load on the gateway has also improved.

Because of the lower usage today, I will do full testing again on Monday – but I already dare to say that the issue is resolved.

Thank you for your valuable advice – it helped a lot.
And I apologize for the delayed replies and maybe some nervousness.

Lukas

Re: HTTPS Inspection significantly slows down loading of images

the_rock — Fri, 09 May 2025 18:37:20 GMT

Awesome job!