Categorize HTTPS Websites

[Nandhakumar]

After I enabled "categorize https websites" on my internet facing gateway, some of the government specific websites stopped working and rest of the internet sites worked well. Any idea what would went wrong?

I haven't seen any drops in smart console during the issue reported time.

[the_rock]

Keep in mind one thing...that setting, in all honesty, is totally useless if you dont do HTTPS inspection. Do you have inspection enabled? Ok, maybe thats wrong term, its not useless, but wont help you much with categorizing anything. If you have url filtering enabled, are you blocking that specific category?

[_Val_]

Not true. With R80.40 and up, SNI enforcement and categorization is used even if HTTPSi is off.

[the_rock]

O really? Hm, I did not know that, my bad...I remember once though working with customer and TAC and they did not have https inspection enabled and TAC guy said in that case categorize https sites was not going to change anything at all. Customer was on R80.40, cant remember what jumbo hotfix though.

[_Val_]

That statement would only be true for R80.30 and below. In R80.40 and up there is SNI verification tech, which sends independent TLS request from a security GW, to verify SNI and a site certificate are matching. It is called sometimes "SSL Lite" and allows us to categorise sites even if HTTPSi is inactive.

We have also publicised this functionality quite a lot in 2019 🙂

[Nandhakumar]

We don't have https inspection enabled. Our gateway is running with Gaia OS R81 and Jumbo Take 23. Then why still its not working?

Can you share what are all the things that will be checked, if we enable "Categorize HTTPS websites" option?

[_Val_]

As already mentioned above, after enabling this option, GW would try retrieving and reading site TLS certificate. Please check your GW can successfully resolve and connect to the mentioned sites directly. curl_cli and nslookup would provide you with the answers. 

[Nandhakumar]

I am not sure what to look specifically in curl_cli output? Can you guide if we need to look specifically something?

Below is the curl command output from the gateway

[Expert@GW:0]# curl_cli https://www.altinn.no
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.


[Expert@GW:0]# curl_cli https://www.virustotal.com/gui/home/upload
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[Expert@GW:0]# curl_cli https://urlcat.checkpoint.com/urlcat/main.htm
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[_Val_]

So that is what I expected. GW cannot validate certificates properly 

[mcatanzaro]

What does the browser show on these now non-functioning websites?

I.e. any specific errors seen?

[Nandhakumar]

No error/drop was seen in smart console logs as well as in browser. These sites have been accessed via SAP application. 

I want to know working principle for "Categorize HTTPS website" option? 

[the_rock]

This post may also be helpful, though I believe this was before R80.40, but phoneboy gave great explanation. I was not aware that those changes @_Val_  mentioned were implemented in newer versions...

https://community.checkpoint.com/t5/Management/Difference-between-HTTPS-Inspection-and-Categorize-HT...

[Nandhakumar]

I am not getting 100% context in that thread. Not sure what might be issue here? How to fix the same?

Does "Categorize HTTPS websites" will inspect encrypted traffic from client machine? If no, then what it will do?

For sure, the site worked well if I un-tick "Categorize HTTPS Websites" option.

Can we enable exception only for specific sites that should not categorize with https?

Also, does it have limitation with SSL site Grade. Only works for sites with the specific grades?

[the_rock]

Maybe opening TAC case would not be a bad idea...below is section from documentation about it:

Application and URL Filtering - Advanced Settings - General (checkpoint.com)

So, it really boils down to whether certificate is trusted or not, because based on that, the categorization would work differently...

Categorize HTTP sites

This option lets Application and URL Filtering assign categories to HTTPS sites without activating HTTPS inspection. It assigns a site category based on its domain name and whether the site has a valid certificate. If the server certificate is:

• Trusted - Application and URL Filtering gets the domain name from the certificate and uses it to categorize the site.

• Not Trusted - Application and URL Filtering assigns a category based on the IP address.

Application and URL Filtering uses these pages (in the SmartConsole Manage & Settings tab > Blades > HTTPS Inspection > Configure in SmartDashboard) to make sure that a certificate is valid:

• Trusted CAs page - Makes sure the certificate is not stolen or revoked.

  Note : If your company issues certificates, you must add your company CA to the list of Trusted CAs.

• HTTPS Validation page - If the certificate is blacklisted, for example, it is not trusted and the site categorized according to its IP address.

  Important - Application and URL Filtering gets the site URL from the SSL "CONNECT" request sent to the proxy, if: There is a proxy between the Firewall and the destination site or The Firewall is configured to work as a proxy

[Timothy_Hall]

A few points:

1) As mentioned R80.40 and later (and R80.20/R80.30 with latest Jumbo HFA) look at the SNI when making a categorization decision instead of just the site name on the certificate, but it is highly unlikely this operation is the cause of your issue.

2) A possible issue could be the problematic sites are forcing TLS 1.3 which is only supported by R81+, but you are using the R81 release so that is unlikely.

3) A little-known fact: When Categorize HTTPS Sites is enabled, the firewall performs certificate verification similarly to what happens when full HTTPS Inspection is enabled.  It is likely that the CA certificate chain for these sites cannot be validated by the firewall (and your curl output seems to suggest this) because it does not have the CA certificate for all CAs in the signing chain utilized by the digital certificates being provided by the problematic sites.  Please provide a screenshot of the settings on Manage & Settings...Blades...HTTPS Inspection...SmartDashboard...HTTPS Inspection...HTTPS Validation, as you may have "Untrusted  server certificate" checked. 

4) Inspect the signing chain for the certificates being presented by the problematic systems, and make sure all entities are included on the "HTTPS Inspection...Trusted CAs" screen in the SmartDashboard.  If they are not it may be as simple as clicking the "Install Now" button at the bottom of this screen to update with the latest CAs, as this update does NOT happen automatically and it is possible the problematic CAs are new and simply not in the trusted list for the firewall, but are for the web browsers.  You may possibly need to manually import the needed chain certificate instead to complete the signing chain properly.  See the following for an example:  sk121615: HTTPS Inspection rejects Netflix traffic

5) Unlikely, but it is possible that the problematic sites are using non-RFC compliant HTTPS, which will suddenly break them when Categorize HTTPS Sites is enabled.  See here: sk109897: How to use "Categorize HTTPS websites" configuration with non-RFC compliant HTTPS traffic

Most of this was mentioned in my IPS/AV/ABOT Video Class which covers HTTPS Inspection...

[Nandhakumar]

Thanks @Timothy_Hall for your response. In our infra, we haven't enabled https inspection. The specific question runs in my mind "how it was working fine for other internet sites except few websites?"

The problematic site is complaint with RFC 5246 (TLS V1.2).  Please find attached screenshot for HTTPS Validation where "Untrusted Server Certificate" was not checked.

Also, does firewall need client machine certificate to be in the trusted CA list in order for outbound traffic to work or no need? (Assume here we enable only 'categorize https websites' option)

[the_rock]

What do you see in the logs if you just do search for url filtering blade when you try accessing those websites? Can you share example of one of the logs?

[mcatanzaro]

Hi,

Your curl output shows as such as you didn't use the '-k' switch for https (ignores warnings/errors in the case you didn't provide the full certificate chain to curl. Make sure you trust the site...).

Do you have an example of one of the domains that is giving you issues?

Tim listed a good number of possibilities of what the issue could be but it would help to have an example site.

[Timothy_Hall]

Yes I realize that you don't have HTTPS Inspection enabled, but when Categorize HTTPS Sites is enabled it performs certificate verification and that is probably what is failing.  The firewall should not need to have the Client cert CA.  Either the traffic to those few websites is not RFC compliant or there is something different about the certificates those sites are using, and I've already mentioned how to investigate these possibilities.  

[genisis__]

Believe it or not I have a similar issue access, of all sites "bbc.co.uk".  We are running R80.40 with JHFA118  and in VSX mode.

I've enabled https inspect on a test VS.  cnn.com works fine and bbc.co.uk fails everytime with https enabled.

I raise a TAC case, months ago and TAC have not provide any resolution.

What I do know is the Global Root CA used is imported into the repository (by default), the intermediates are not there (yes did try to import this as well and TAC were on a zoom with me) but that should not really matter.

The only way I could get this working was to import the actually 'bbc.co.uk' certificate, which is totally wrong (TAC have also seen this).

For me using https inspection is completely useless with Checkpoint  for two reason:

- When we have a site that does not work, the evidence so far indicates that Checkpoint cannot resolve it within a reasonable timeframe.

- The resource requirements to use https inspection is just not financially viable, its better to use a cheaper competitor that does this at a fraction of the cost and with dedicate inbuild https inspection module without the hefty price tag.

Sorry this sounds like a rant, but unfortunately this has been my experience.

also one odd thing that TAC mentioned which I've challenged:

Apparently VS0 needs access to the internet for OSCP access, despite the fact I see no traffic from VS0 even attempting this.

[the_rock]

Sadly, I have to agree. I work with one customer who has inspection enabled and had spent so many hours with TAC on the phone trying to make it work correctly. Even now, so many moths later, its not working 100% as it should. Personally, I wish it worked more flawlessly and that there was a bit better support when there is a problem.

[Wolfgang]

@genisis__  I know that some features are not working perfect if newly available. But since R80.40 HTTPS-inspection in my view is working fine. You have to tune and sure there are some applications and websites to bypass (as an example some conferencing tools or websites requesting client certificates). But with other solutions you have to do the same. 

I tried your named website https://bbc.co.uk  without problems on a R80.40 and R81 system with enabled HTTPS inspection and both are inspected and shown without problems.

But I agree with you, TAC should solve these problems and my experience is there are not enough engineers available with the needed know how for troubleshooting such complex problems.

[genisis__]

If Checkpoint ever solve the mystery then will feed back here; also lets hope TAC is better in 2022..hint hint 😉

[_Val_]

@genisis__ @the_rock just a few point from my end:

1. This thread is an off-topic and is not related to the issue reported by the topic-starter.

2. I certainly support a right to vent in the community, but hijacking someone else's discussion for that may not be the best way.

3. @genisis__, I have failed to find any case in your list of SR that would be relevant. If you want a second opinion, please send me the SR# via PM. From what you have described above, it seems like the certificate chain validation fails. if you want, we can discuss this in a separate post.

Thanks for your understanding. 

[genisis__]

Will do

[the_rock]

Thats fair. @Nandhakumar ...back to your actual issue...have you ran log filter just for blade:"URL Filtering" or blade:"Https inspection" to see if it gives more details?

[Nandhakumar]

To all experts here...

I am expecting crystal clear answer from your side rather than assumptions. Following are my questions, please answer my question and provide your suggestions.

 What is the major difference between HTTPS inspection and Categorize HTTPS Websites?

Could see HTTPS Inspection policy added automatically from R80.40 version even though we are not using it, Any reason? What it will do in non use cases?

If we enable only "Categorize HTTPS Websites" option, why some of the specific websites alone not working and rest of internet traffic works good? What we have to look specifically here?

If the website uses self signed certificate, will "Categorize HTTPS Websites" block traffic (Consider here certificate is imported in HTTPS inspection dashboard)?

At least the site has 3 certificates, 1st is its own, 2nd is intermediate and 3rd is root. In this case, import of root certificate on HTTPS Dashboard alone is fine?

How long will it take to be in affect after successful import of certificate and policy push?

Final question, do we need to import certificate and all other steps if we have enabled only Categorize HTTPS websites option but not HTTPS inspection?

[_Val_]

I believe you already have most of the answers up above. Yet, it does not heart much to repeat those.

HTTPS Inspection means GW is terminating TLS connections, decrypts them on a client side and re-encrypt to the server. This allows inspecting the data flow, applications behaviour, and content sent through TLS tunnel with multiple blades: IPS, AVI, AB, etc.

HTTPS URL categorization uses  a special mechanism of SNI verification, when a GW sends and independent HTTPS request to the same server and thus is able to get clear text read on encrypted URL, to categorise it.

Enabling HTTPS categorization does not turn on your HTTPS Inspection and in fact works out of the box with R80.40 and up.

TLS process requires validation of Web server certificate through CRL and certification chain. On top, root CA signing certificates should be in the list of trusted CAs to be accepted.

We have figured out already that in your case three failing sites have different issues with the certificates. That by itself should not cause a failure with HTTPS inspection or with HTTPS categorisation, but it gives you a direction to continue the investigation.

It is most likely that your AC and URL filtering is not configured properly. It either has "Hold" action for categorization, or you miss a cleanup rule for non-categorized sites in the policy. From where I stand, categorization of those three sites is not complete (HOLD action and consequence failure of the traffic), or is categorized as "Unknown" but is not matched on any accept rule. 

If all above does not make sense to you, or if you do not know how to progress further with the troubleshooting, open a TAC case, to get expert assistance.

Just a reminder, CheckMates is a great tool for knowledge sharing and for technical enquiries, but it is not a support tool. 

[Nandhakumar]

Thanks all for your response, I have got some idea with it. I already raised case with vendor 

Regarding your points about curl command output.. Those all are not non-working sites.. If you look at the output of https://urlcat.checkpoint.com/urlcat/main.htm, it is a checkpoint website. So issue might be something different.

We have hold option enabled but it will categorize based on URL categorization database, if I am not wrong. We have explicit cleanup rule defined and we are not blocking uncategorized sites in policy explicitly.  According to Checkpoint URL categorization site is categorized under Government/Military category.

Let see, How deep TAC will dive into this?

[the_rock]

@_Val_ ...thats PERFECT explanation brother!!