Windows Update Services with HTTPS inspection enabled

[lucafabbri365]

Hello,
we are having issues accessing Windows Update with HTTPs Inspection enabled (Check Point R80.20 with Take 87) and "Bypass HTTPS inspection of traffic to well-known software update services" option checked.

If, from browser, I try to surf to https://slscr.update.microsoft.com, instead of getting "403 - Forbidden: Access is denied.", I get the "ERR_CONNECTION_RESET" error.

Any advice ?

Thank you,

Luca

[Kim_Moberg]

Hi Luca

I am seeing the similar issues while running R80.20 Take 80.

it was blocking windows update for Windows 10 Ver 1903 while doing https inspection but as soon I am using an uninspected subnet it worked.

on the working subnet I was usning wireshark to search and filter for Server HELLO messages to find domains which Windows update CDN (Content Domain Network) was being used by it.

I am though not 100% through yet.. because I have bypass the following hosts on layer 7 but also tried to bypass on ip on layer 4 in the OSI model. 

The hosts are which also include some other Microsoft services:

ams15s32-in-f3\.1e100\.net
wdcp\.microsoft\.com
wns\.windows\.com
wdcpalt\.microsoft\.com
update\.microsoft\.com
download\.microsoft\.com
windowsupdate\.microsoft\.com
download\.windowsupdate\.com
wustat\.windows\.com
ntservicepack\.microsoft\.com
stats\.microsoft\.com
wns\.windows\.com
nexus\.officeapps\.live\.com
fe2\.update\.microsoft\.com
delivery\.mp\.microsoft\.com
vortex-win\.data\.microsoft\.com
cp601-prod\.do\.dsp\.mp\.microsoft\.com
geover-prod\.do\.dsp\.mp\.microsoft\.com
big\.telemetry\.microsoft\.com
ctldl\.windowsupdate\.com
audownload\.windowsupdate\.nsatc\.net
au\.download\.windowsupdate\.com\.hwcdn\.net
slscr\.update\.microsoft\.com
sfdataservice\.microsoft\.com
windowsupdate\.com
windows\.com
slscr\.update\.microsoft\.com
slscr\.update\.microosft\.com\.akadns\.net
v10\.events\.data\.microsoft\.com
v10\.event\.data\.microsoft\.com\.aria\.akadns\.net
onecollector\.cloudapp\.aria\.akadns\.net
fe2cr\.update\.microsoft\.com
fe2cr\.update\.microsoft\.com\.akadns\.net

Did you create a TAC on this issue?

Am I missing some host to get it to work? 

@HeikoAnkenbrand do you have any experience with this issue?

Thanks

[lucafabbri365]

Hello @Kim_Moberg,

thank you for your reply. I didn't create any TAC yet; I just shared the problem here for finding any useful information before going to open a support ticket.

We have different behavior, depending on Windows 10 build version, but in all cases, they cannot access Windows Update services. I didn't find an official Microsoft documentation listing all URLs used by Software Updates, but only:

1. Windows 10, version 1903, connection endpoints for non-Enterprise editions (not only SU)
2. Step 2: Configure WSUS (it is more related to WSUS)

I tried to create a bypass rule containing all URLs listed in the article at point 2, but it doesn't work either.
Maybe creating a bypass rule containing IP address could solve the issue, but it isn't so flexible (they could change frequently).

At this point, if no other has a solution to this, I can proceed to open a TAC.

Bye,
Luca

[lucafabbri365]

Update: I opened a TAC. I'll give you more information asap.

[Leonardo_Tessar]

Do you have any update about this issue?

[lucafabbri365]

Hello @Leonardo_Tessar,
I have just a quick update regarding this issue (not yet solved).

First of all, support asked me to manual update the Trusted CA list; for some reason, it wasn't up-to-date, even if the option "Notify when a Trusted CA and Blacklist update file is available for installation" on SmartDashboard is selected:

1. sk132812 steps 1 - 4 to force list update (if you have doubts that the list updated).
2. Check the list at  /opt/CPshrd-R80.20/database/downloads/TRUSTED_CA/2.0/2.3/updateFile.zip is up to date 
3. Download the list to admin PC
4. Upload it manually through SmartDashboard
5. Install Policy

The updateFile.zip at point 2 doesn't update, file timestamp was old May 13 (a deep investigation will be needed). So they provide me the file via SFTP and after uploaded it manually on SmartDashboard (points 4 - 5), about 94 Trusted CA were updated and 3 removed.

However, it didn't resolve the issue. I'm waiting for further steps.

Bye,
Luca

[Kim_Moberg]

Hi Luca

I also created a TAC and received the same step-by-step to update the root certificate list.

Still windows update services doesn't work after the suggested solution and waiting for R&D to update the next step.

[Kim_Moberg]

An short update.
Check Point have managed to reproduce this issue in their labs and RnD currently working on it..
Looking forward to see an soon resolution on this issue. 🙂

[lucafabbri365]

Hello @Kim_Moberg,

thank you for the update.
I didn't get any feedback yet - maybe they are waiting for feedback from R&D.

Bye,
Luca

[FedericoMeiners]

Hello,

@Kim_Moberg When you used Wireshark did you only the client hello going and then a connection reset? 

If you can please try to enable the probe bypass feature to check if that solves the issue? You can enable this on the fly, try this on non business hour to avoid possible SNI issues

None of these commands requires a reboot or cpstop/cpstart
Enable on the fly
[Expert@HostName]# fw ctl set int enhanced_ssl_inspection 1
Disable on the fly
[Expert@HostName]# fw ctl set int enhanced_ssl_inspection 0

If it's not working it's maybe because probe bypass is working on fail close, you can change this to fail open with the following:
[Expert@HostName]#fw ctl set int bypass_on_enhanced_ssl_inspection 1
To change it back to normal:
[Expert@HostName]#fw ctl set int bypass_on_enhanced_ssl_inspection 0

If you want more information you can check sk104717 - Improvements in HTTPS Inspection Bypass mechanism - Probe Bypass

Regards,

[lucafabbri365]

Hello @FedericoMeiners ,

thank you for your feedback. I'll try this late evening.

Just to add this other information: before switching between probe bypass enabled/disabled on-the-fly, I usually clear the CN cache with command fw tab -t cptls_server_cn_cache -x -y.

This to avoid any "false" results. The support suggested to me that command in an old ticket (related to Check Point R80.10).

Regards,
Luca

[Ilya_Yusupov]

Hi all,

we are replicate the issue and working with RnD to solve it, as current WA you will need to bypass the domains in the SSLi RB.

This issue only happen on Windows 10 clients.

i will update once we will have resolution.

Thanks,

Ilya 

[lucafabbri365]

Hello @Ilya_Yusupov,

thank you for your feedback.

I'm pretty ignorant, so, please, can you explain to me what "SSLi RB" means? I can imagine it has to do with SSL; some by-pass rule ?

Thank you again.

Regards,
Luca

[Ilya_Yusupov]

@lucafabbri365  - i will contact you offline.

[Kim_Moberg]

@lucafabbri365 and @Ilya_Yusupov 

R&D were able to reproduce this error and found out Microsoft changed its servers to support only ECDSA cipher suites.
Firewall by default, does not propose ECDSA

You have to enabled ECDSA and update https_inspection_white_list.bin

Please contact TAC to get the solution. It worked for me on R80.20 AC

[lucafabbri365]

Hello @Kim_Moberg ,
I received feedback from support with the same procedure.

I'll update you, here, asap.

Bye,
Luca

[Kim_Moberg]

@lucafabbri365 

Good to hear.

You might experience the command cpstop;cpstart fails and your cluster sync dont work.

then go once again with cpstart and you cluster will be running again..

I experienced this on both members in my cluster after the update. So cpstart is way to go 🙂

Let me know how the result is.

[lucafabbri365]

Hello @Kim_Moberg,
I was thinking, instead of cpstop and cpstart, to reboot each node.

However I'll try them at first.

Bye,
Luca

[Peter_Lyndley]

I assume there will be a public SK for this shortly ? so that everyone else knows what is going on?

[lucafabbri365]

Hello all,
just a quick update:

1. Into one Check Point node, we enabled ECDSA and update https_inspection_white_list.bin (as suggested by support)
2. Into the other node, we enabled ECDSA only (as discussed and suggested by @Ilya_Yusupov, privately)

Both solutions seem to work. So it shouldn't need to replace *.bin file.

I'm waiting for confirmation.

Bye,
Luca

[JuanMora]

Please confirm me what was the solution or some SK?

[avega]

 sk162517: Windows 10 update fails when HTTPS Inspection enabled on Security Gateway