R80.30 - Recent HTTPS inspection issues

[Howard_Gyton]

For the last few months we had been seeing a steady increase in sites that did not suffer undergoing HTTPS inspection issues very well, and a small handful that we could not even create reliable "Bypass" rules for.

Our upgrade path was from R77.30 to R80.30 on the gateways, and an export/import to our new R80.30 management server.

We had been given a number of options by both CP and our support partners, all of which sadly failed, but I and a colleague dug a little deeper, and found the real cause and fix.

One of the sites in question was https://roccochiou.weebly.com/

When attempting to connect to this site, due to some local settings I will come to, a TLS1.1 connection was attempted, at the TLS Record layer (outer for simplicity) but TLS1.2 at the Client Hello layer (Inner).

Due to a known issue, documented in https://github.com/openssl/openssl/blob/OpenSSL_1_1_1-stable/ssl/record/rec_layer_s3.c#L850, this would cause the site to fail to load.

Advice given was to rebuild our firewalls to R80.30, which we didn't and would have failed anyway, and adding the sites certificate/chain, to the firewall list of trusted CA's, which also didn't work.

What it turned out to be in the end was an SK we followed back in 2017, when we were still running R77.30.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

In essence, this was to enable enchanced SSL inpspection and also raise the "ssl_min_ver" to TLS1.1.

During a more recent ticket, again with sites failing to load, we were advised to turn off the enhanced SSL inspection issue, but we had forgotten about the "tsl_min_ver".  I just happened to run a search for "HTTPS Inspection TLS1.1" on this forum, and found an article that jogged my memory, and solved the issue for us by changing the "ssl_min_ver" back to its default value of "TLS1.0"

My take on this, on reflection, is that the ticket had dragged on for weeks with no end in site, had repeatedly resisted our requests for escalation, and overall could have been handled better.  One bit of advice we thought of, as part of support staff's scripts would be:

1. Ask the customer whether they have any non-default values set on relevant tables.

2. Possibly have a script that would trawl a customers database for non-default values, then inspect tables relevant to the issue at hand.

With hindsight being 20:20 its easy to make these suggestions, but hopefully they would speed up ticket resolution, and would be applicable to other issues, not just HTTPS Inspection.

Howard

[FedericoMeiners]

Howard,

Thanks for the useful information about the issues caused for non default TLS configurations.

Unluckily there is no proper way to properly bypass traffic, the best "way" to do this is to not specify the network prefix on the HTTPS Inspection policy. You can read more about this in my post: Outbound SSL Inspection: A war story 

Also as you could see is really hard to troubleshoot TLS Inspection issues, most of the time you don't get useful information out of the debugs.

After all these issues caused with HTTPS Inspection in general (Not only in Check Point) you can imagine why almost no companies use it 😀

I've been told that there were some major improvements on R80.40 regarding TLS Inspection engine, but I didn't manage to try it yet.

Regards,

[Howard_Gyton]

One of the issues we had with this configuration, and the very few sites that were actually affected, was the logs never reflected the communications to the affected site correctly.  The traffic was never logged as either being inspected or bypassed, with the only record of it being a firewall allow event.  After the change to "min_ssl_ver", traffic is being logged correctly, as is site classification, so bypass if needed would work.

One thing I forgot to mention.  We have a number of sub interfaces that hang off the internal 10Gbit bonded pair.  One of these is for wireless access, using Eduroam.

On this network the affected sites loaded okay, which is configured to bypass all HTTPS inspection, and the only reason it worked here, we believe, is because it is on a sub-interface.  We had limited success with emulating this on other networks of the same configuration, but certainly any network that existed on a core network and not a sub-interface didn't work.

Essentially what we observed was the "tls_min_ver" setting of "TLS1.1", in our database, was ignored on those sub interfaces. Traffic on internal networks had the TLS Record Layer set at "TLS1.1", and the site failed to load.  On a network on a sub-interface of the firewall, the TLS Record Layer was set to "TLS1.0", which it should not be, and the site loaded.  This shouldn't have happened.

I wonder if this is a bug, albeit one that worked in our favour, and assisted in determining the root cause?

Anyone from Check Point care to speculate here, as that doesn't seem right to me.

Howard

[Ilya_Yusupov]

@Howard_Gyton  - Let's continue it offline.

[Daniel_Taney]

I was just reading through this chain and wanted to clarify your situation.... are you saying you did upgrade to R80.30 and found  that the hot fix described in sk107744 (which claims to have been fixed in R80.10+) did not resolve the issue that existed in R77.30 and you had to fallback to adjusting the ssl_min_ver value?

Or was it just that having this modified value carried over from an older installation caused unpredictable results in a newer release? Sorry for all the questions, but I am looking to begin testing HTTPS inspection in a lab environment and have been trying to keep an eye on issues others experience.

Thanks!

[Howard_Gyton]

Hi Daniel,

We applied sk107744 when running R77.30 back in 2017.  When we upgraded to R80.30 back in November last year, this was in place.  Enchanced SSL inspection was left on, as was "TLS1.1" for "ssl_min_ver".  This seemed to have undesirable effects under R80.30, and the former was turned off, but not the latter till last week.

With both turned off, or rather reverted to default values, then affected sides loaded correctly.  The oddity we observed on the back of this was that sub-interfaces of the firewall seemed to ignore the non-default "ssl_min_ver" value, and hosts on that network continued to use "TLS1.0" for some reason.

I believe this has now been confirmed by QA.

Howard

[Daniel_Taney]

Ok, good to know! Thanks for clarifying!

[doraskayo]

Hi,

I'd like to shed some light on the root cause for the issue and our plans around it.

As you found, this is indeed an in issue in the web server's implementation of the TLS 1.2 protocol. The server is unable to handle the case that the record layer version is set to TLS 1.1 and terminates the connections.

HTTPS Inspection as a TLS client sets the record layer version to TLS 1.1 (0x0302) in the Client Hello message when the minimum TLS version is set to TLS 1.1 through configuration. The TLS 1.2 RFC discusses this record layer version in section E.1.:

TLS clients that wish to negotiate with older servers MAY send any
value {03,XX} as the record layer version number.  Typical values
would be {03,00}, the lowest version number supported by the client,
and the value of ClientHello.client_version.  No single value will
guarantee interoperability with all old servers, but this is a
complex topic beyond the scope of this document.

---

The RFC proposes three possible values for the record layer version field:

1. “{03,00}” (SSLv3)
2. “the value in ClientHello.client_version.”
3. “the lowest version number supported by the client”

It then states that implementations may choose one of the options, but none of them guarantee compatibility with older servers. In our implementation we chose option number 3.

Since the RFC proposes three possible values for this field, it is expected from servers that implement it to handle them all three in a graceful manner. Unfortunately, this isn't always the case, as we see with the server you mentioned.

We're currently assessing whether we can work around this issue in our implementation without creating additional compatibility issues. In the meantime, the workaround for this issue is to set the minimum TLS version to either TLS 1.0 or TLS 1.2, depending on your security requirements.

Thanks,
Dor