HTTPS Inspection (issues, exceptions, etc.)

[marcyn]

Hi CheckMates,

HTTPS Inspection ... hmm these days this is a must have feature - no question about it 🙂
But as all of us know there are situations when some page doesn't work... and in 99% of such cases admin just makes an exception with bypass.

I would like to dig a little bit more ... and see why such page doesn't work so I thought about some debugs...

I've found that in lab env. I can have 100% reproductable issue with "dev Windows 11" machine in HyperV where if HTTPS Inspection is enabled Microsoft Store app will not load. And I know that also Microsoft Teams app doesn't allow users to log in (if they are already logged in it works well ... funny thing is that Teams as web application works well ... and this "issue" with "thick app" only exists on Windows 11 ... on Windows 10 everything is OK).

So....
I wanted to dig a little big more and I see that when we have HTTPS Inspection enabled and Inspect rule (no bypass rule) it is Microsoft site that "doesn't like us" because in PCAP I see that after TLS Handshake it sends "Encrypted Alert". After that Quantum sends RST Flag.

Unfortunately this is .... encrypted ... so I can't see what exactly Microsoft site sends ... and what it doesn't like.
So ... maybe some debugs will tell me what is inside ?

Unfortunately I didn't find a way to get such info...
Debug of WSTLSD doesn't show anything interesting (even with TDERROR_ALL_ALL=10).... all certs are fine, every validation is OK, etc. ....
/sk105559/

How to find what's inside this "Encrypted Alert" ?
This is not 100% sure that it will be error, it could also be information that there is no need to send more and we can send application data from now. But..... because of this RST I assume that is error message indead 🙂

Sure, I can deal with this isse with bypass rule for IP 23.200.161.193 ... but what if such IP is constantly changing ?
You will say "Updatable object" ... sure - but what if there is no such Updaptable object that match this case ?
Maybe "URL Filtering" then .... and again what if there is also no match ?

In my opinion it would be best if I could find what exactly the other site doesn't like - maybe having such info I would be able to find solution ?

I'm curious what do you thing about it, maybe somebody found a way to deal with this issue ?

--
Best
m.

[the_rock]

I would be totally open to test this in my fully working https inspection lab (latest R81.20 jumbo as well). Can you please give me the scenario, just to make sure Im not missing anything...are you saying that when httpsi is enabled on the gateway, Microsoft store does not work?

If so, that would be super easy for me to confirm.

Kind regards,

Andy

[marcyn]

Hi Andy,

Yes, it should be extremely easy to reproduce it.

Scenario could be like this:

1) install dev Windows 11 in HyperV (~20min)

2) import Check Point's ssli CA to this machine (~2min)

3) open Microsoft Store (~1min)

4) if for any reason Microsoft Store will open without an error code like 0x800... try to search for something

I was able to reproduce this in different env. one with R81.10, one with R81.20.

But ... this case was created not to confirm that it works like that... but to find a solution. To find if anybody knows how to see "decrypted" TLS that would tell what is inside this "Encrypted Alert" 😀

I don't see any other solution besides bypass rule .... but as I described it is not always as easy to do it as it looks like 😀

Anyway Andy, if you will confirm that it will mean that I'm not lying 😀 

m.

[the_rock]

Come on brother, of course you are not lying, I believe you 100% : - ). Anyway, can you send a screenshot of the https inspection policy if you dont mind? I have environment ready.

Andy

[marcyn]

Hi Andy,

Default one 🙂

One rule: any -> Internet = inspect

No bypass rules

Pure lab environment 🙂

m.

[the_rock]

Ok, got it. Give me some time, gotta finish my daily exercise (important to keep healthy) and will check and report.

Kind regards,

Andy

[the_rock]

Just tested it, works fine in my lab, windows update and windows store, MS teams as well. Can you ensure below settings in attached doc are the same?

Kind regards,

Andy

[marcyn]

Interesting 😀

It's late here and I don't have Access to my lab now... I will update you in a few hours.

Thanks

m.

[the_rock]

When I was in my 20s, late was 3 am lol. Now, in my 40s, sometimes even 9 pm feels that way HAHA

Np my friend, happy to do remote and Im sure we can fix it when you are available.

Have a great night 🙂

Andy

[marcyn]

Hi Andy,

Hmmm it looks like it was not late enought ... for Check Point's troubleshooting 🤣

Regarding ssli settings - I see that probably you have default one as I have ... at least those in SmartDashboard.

Mine are 100% default - as you can see on attached PDF.
It's really interesting that we have different results 🙂

Windows 11 is also "out of the box in HyperV" - network configured, CA imported ... that's it, nothing more 🙂

But anyway ... question remains the same "how to see what's inside this Encrypted Alert" ?

BTW
This setting to Bypass whitelisted apps in SmartDashboard in my opinion has nothing in common with Microsoft Store and Microsoft Teams ... but yeah for Windows Update and other update services.

BTW2
Of course once I disable ssli ... everything works fine 🙂

BTW3
Now I can go sleep ... 🙂

BTW4
In this lab env Teams works fine ... on 2nd lab I was not able to sign in with similiar error... without ssli no issues at all.
Looking on PCAP  .... the same "Encrypted Alert".

m.

[the_rock]

Excellent work mate, we can check Monday, get some rest.

Kind regards,

Andy

[Tobias_Moritz]

The usual root cause is simply:

The application you have problems with HTTPS inspection has a hardcoded trust and therefor does not accept the CA certificate your HTTPS inspection gateway is using.

This can be a certificate public key pinning (unusual) or a CA public key pinning (more likly).

A well-known service protected by this is the Entra ID authentication service (former Azure AD) when used by specific applications like MS Outlook or Teams. This is why your MS teams rich client is working with HTTPS Inspection when login was done without inspection. The bearer token is still valid so no authentication needed and only authentication is protected by pinning.

What can you do to fix it (without bypass)? Nothing. The issue that HTTPS inspection is not working here is by design. It doesn't matter which HTTPS inspection solution you are using. You will have the same problem with local solutions like Fiddler, when you enable HTTPS inspection there. So even when importing a specific server cert in the client OS local trust store (thats how fiddler is doing it, not just importing the CA), it will not work. You also have to enter login.microsoft.com in the fiddler HTTPS inspection exception list.

That MS teams authentication is working in your browser has also a simple reason: Your browser does not use PK pinning for that site (yet).

Check Point has entered some well-known FQDNs which are affected by PK pinning applications in the update object "HTTPS services - recommended bypass".

And regarding your packet capture: You cannot see that problem reason in a packet capture. Its not the gateway, or server that terminates the connection here, its the client. The RST you see from gateway is after client has closed the connection with FIN,ACK and it is send because the gateway closes the connection afterwards and not waits for the server to send its FIN,ACK.

Regarding that encrypted alert: "Encrypted Alert" means Wireshark can't decrypt it. The reason why this packet appears may vary, but if it appears just before a TCP FIN. like in your capture, it is most likly a "close_notify".

Regarding what to enter into bypass list: Just create a custom application object for that.

[marcyn]

Hi Tobias_Moritz,

To be honest I completely agree with all what you wrote.
Especially with this "What can you do to fix it (without bypass)? Nothing". I also think that the only solution here is to make bypass rule because if we have key pinning .... there is nothing more we can do.

Unfortunately all of the bypass attempts doesn't work 🙂
1) Category/Custom Application: Microsoft & Office365 Services, etc
2) multiple Microsoft domains as FQDN (.live.com, .microsoft.com, .microsoftonline.com, etc, etc)
3) HTTPS services - recommended bypass

Because of the above I assume that Microsoft uses here some other hosts/domains/etc that are not included in Check Point's objects.

Regarding "Encrypted Alert" - as I wrote it doesn't need to be error ... it could be like you wrote simply notification.

Regarding my PCAP screenshot - 192.168.137.101 is my Check Point's external address (yeah it is not "real external IP" but for my Check Point it is external) - so Check Point sends FIN flag ... then Microsoft confirms it, then Check Point sends RST.
Because it is soon after it receives this "Encrypted Alert" I assumed that maybe there is something "important" that Microsoft sends after FIN.

Anyway ... it still doesn't work for me regarding any bypass attempts I try ... and to be honest I don't have any other options here to try in my mind.

Below you will see all attempts:

 

I also think that bypass rule is the only solution here .... but if above three doesn't work ... which one will ? 🙂

m.

[the_rock]

Im sure if we do remote session, we can figure this out.

Andy

[Wolfgang]

@marcyn all of your shown rules are configured with „inspect“ not „bypass“.

[marcyn]

Hi Wolfgang,

Yes ... and No...because they are negated.

So it really means inspect all the rest except of these settings 😀

But yeah it could be without negate and with action set as bypass.

Or maybe if it is not bypass explicitly... it will not work. I mean that for me "negate inspect = bypass". Easy to check, I will do it later.

In a matter of fact I personally like to have bypass ... it's much clearer. But these negate "version" is from my Customer ... and it looked good to me, so I even didn't checked if !inspect=bypass ... because it was captain obvious for me 😀

m.

[Wolfgang]

Yes this is negated, but there is no other rule shown to inspect or bypass the rest of the traffic. Negating a source does not mean you negated „inspect“.

I would prefer a rule with explicit bypass for your shown sources. Negated fields are are a good thing but always stresses my brain under pressure to really understand what‘s going on if something negated.

[the_rock]

If you want me to test anything in my lab, please share, happy to do it, not an issue mate.

Cheers,

Andy

[marcyn]

Hi guys,

@the_rock, @Wolfgang , @Tobias_Moritz thank you for all of your comments here.

The reason why I created this case on the first place was to know if somebody know the way to see decrypted TLS on the CheckPoint<->server perspective ... because of course on the perspective client<->CheckPoint it's no problem at all.
Encrypted Alert was just as an example here...
Unfortunately debugs of WSTLSD process are not helpful in this case because we can say that they only help us with understanding is certificates chains, validity, etc are fine ... I dare to say that nothing more 🙂

And there way also 2nd reason ... to see if anybody know other way then bypass ... but it seems that you sonfirmed that there is none.

So to summarize:
1) no more lab env. tests ... it's time to see that in production env. of my Customer ... that's it !
2) Microsoft Store issue in my lab env. was just a coincedence ... the main issue is with Microsoft Teams in production env. of my Customer ... unfortunately it worked fine in my lab env. so I was not able to reproduce this issue (so did Andy ... in his lab it was fine too). It could confirm that no bypass should by neccessary for this ... by we will see 🙂

So again, thank you guys for your input here !
It's time to see the real life ... and find the correct bypass rule 🙂

@Wolfgang I completly agree with you with this negate/inspect/bypass perspective. I would also made simple bypass rule for this.
Customer likes to make it with negate way ... and that's also fine if this is better for him.
Rules with negate inspect means bypass ... this particular case, inspect all the rest - so if he prefers this way, be my guest 🙂

I will let you know after remote session with Customer on production env. what finally helped here.
Thanks.

m.

[Peter_Lyndley · Accepted solution]

We have come across exactly the same thing, where HTTPS Inspection is enabled for login.microsoftonline.com and a customer either fresh installs or upgrades to Windows 11.... Windows cannot do its FIRST logon.

Once the first logon is done then everything is fine, so there (i assume) is some pinning or similar with the first logon. Once the initial login is done then inspection can be turned back on without incident.

we ALSO found that applying the latest JHF (110) on R81.10 also fixed the issue !! I could not see anything in the public notes that would match this but we have proved it across 50 firewalls that indeed JHF110 fixes the issue of first logon.

Hope this info is useful to others too

Pete

[marcyn]

Hi Pete,

Interesting, thank you for this input.

On friday I will see on remote session how it will go.

m.

[marcyn]

Hi,

It took quite a long time ... but I can finally confirm that @Peter_Lyndley's suggestion was the good one - after update to R81.20 (Customer didn't have R81.10 Take 110 so it was better to update directly to R81.20) issue is gone.

So in case somebody else would have the same issue - update to at least R81.10 Take 110.

--
Best
m.

[the_rock]

I also found that ever since R81.20 came out, there were lots of https inspection improvements.

Best,

Andy