HTTPS inspection in R77.30 and R80.10 with various combinations of proxy and probe bypass

[Albert_Wilkes]

Hi,

I have done some testing on behalf of a customer who has had issues with HTTPS inspection. Based on this I thought it would make sense to write up my findings for my colleagues in our CP support team as well as submit them for everyone interested to confirm or disprove.

-

spoiler alert: Since the latest JHF in R80.20 as well as R80.30 and newer, CP uses the SNI in direction client-to-server for determining the hostname of the request which is much better than using the server certificate. This article will only cover versions prior to this and thus will be obsolete for most users of R80.20 and up

-

The tests performed are meant to show the real-life effects and benefits of using HTTPS inspection with and without (CP or a 3rd party system in a DMZ as) a proxy as well as (not) enabling probe bypass. The firewall used was R77.30 open server with a relatively recent JHF, I believe it was 301. I since have tested this on R80.10 and the behaviour seems to be identical.

Lets get started with it…

1. Using the CP as a proxy or having a proxy in a DMZ

This might be new for some – it certainly surprised me - but logging is much better when using a proxy – either “behind” the CP e.g. in a DMZ or using the CP itself as a proxy. The below tests were done with a website www.uvex-safety.co.uk

This is the situation when CP is only seeing the SSL connection without a proxy request:

 

Note that some relevant but far from exact information is in the right-hand-side column which is “Resource”. The “Destination” on the LHS column is rather distracting and near irrelevant at best.

For the non-proxied request logging you get the result of reverse lookup in the Destination column (see above screenshot) while the resource column is containing just the MAIN CN only (see screenshot below) but not the actual website that the user went to – which is one of dozens of alternate names in the certificate.

 

Compare this with the situation when the CP is used as a proxy:

 

Concluding: when going to a website like the www.uvex-safety.co.uk website with subject alternate names, you’ll only find www.uvex-safety.co.uk in the CP logs when you use the CP as a proxy – the destination in this case is the CP itself and the resource column contains the interesting data.

Another good reason for considering the use of “a” proxy (may it be the CP or not), is that it fixes some HTTPS inspection incompatibilities: See sk92888: “If a proxy server is used in the organization, place the Security Gateway between the end user and the proxy (instead of placing it between the proxy and the Internet). This will allow Bypass rules based on category to work. “

I have found that the blocking works regardless when using www.uvex-safety.co.uk as the URL in URL filtering but as per that SK you’ll need a proxy request to arrive at or traverse the firewall for category actions to work.

2. Using the CP with probe bypass AKA enhanced_ssl_inspection

Please note that other forum articles seem to indicate that there will be enhancements to probe bypass for later versions of R80.10. See this thread: https://community.checkpoint.com/message/18249-re-is-there-any-workaround-for-sni-https-traffic-when...  

Enhanced_ssl_inspection has a severe limitation so the decision to turn on this parameter should probably not be taken lightly. See the section from sk104717 at the very bottom of this post. To put it into simple words the default setting enhanced_ssl_inspection 0 can or will break

• Non-Browser Applications
• Client certs
• First connections even if bypassed

Which is why CP probably came up with the enhanced_ssl_inspection aka probe bypass…. Now lets look at this enhanced mechanism, surely “enhanced” means “better” and the single limitation seems innocuous enough to just turn on the enhanced mode/probe bypass:

• HTTPS Inspection will not work for sites that require SNI extension in the SSL "Client hello" packet.

It sounds innocuous because most people don’t come across SNI frequently unless they host many websites themselves. SNI is being used by big webhosters that host multiple webpages on one IP and can’t or don’t want to put all the domains hosted on that one IP into one certificate  – think amazon AWS and the like. As per https://blogs.akamai.com/2017/03/reaching-toward-universal-tls-sni.html , more than 30% of the top 100.000 websites are (on HTTPS) SNI only.

This translates into two things:

• All HTTPS accesses that traverse the firewall towards an IP address that is hosted at a big webhoster with HTTPS need bypassing BY IP
• By the nature of bypassing access to a website that requires SNI (probably because a webhoster has multiple webpages on the IP) it will mean that you’ll bypass not just the very website that you’ve resolved to an IP but also an unknown number of other ones

The first point means this must be a difficult scenario to manage and the second means it is insecure on top of this.

Solution:

It seems that both using a proxy request to or through the CP (see the top section of this post) or disabling enhanced_ssl_inspection allow access to the site. This time the tests were done against another webpage but with the same firewall as per the top of this post. This time the URL is prescription.uvex-safety.co.uk. As opposed to the www.uvex-safety.co.uk website, this is not using subject alternative names but SNI instead probably because it is hosted with lots of other URLs on an Amazon IP.

At the bottom of this post you see screenshots of the behaviour when probe bypass is enabled in these two screenshots:

Client side:

On the internal side you only ever see syn requests. The firewall learns the destination IP the connection is going to go to and sends its own SYN and SSL negotiation on the server side:  

Server side:

On the external side you see SSLv3 Client Hello going out, followed by a reset from the server as the firewall couldn’t learn the hostname at this point yet as the client connection has “only” proceeded to the SYN. While that SYN contains the destination IP, it doesn’t contain the hostname, hence no chance to do SNI on the outside and indicate to the webserver what page the client wants to request.

The next screenshot shows the behaviour when probe bypass is disabled: There is an SNI packet that actually gets sent across.

 

I’ve repeated the tests with proxy and probe bypass enabled as well and they were successful so this is another reason to consider proxy requests over straight tunnels to the HTTPS server.

PS: The most bizarre thing for me was the fact that you would also get connectivity when you DON’T have any bypasses configured as per the APCL example below – when I had enable even non-matching bypasses, access wouldn’t work.

I guess the CP didn't need to wait for the server to provide any information as to whether the first 2 policies matched and just went ahead sending the traffic on as there were no URL or category based bypasses so basically it didn't need doing the probe bypass, even though it was technically enabled.

PPS: When you test this yourself, note that there seems to be some cache on the CP so when you transition from off to on, the sites don’t immediately fail, you’ll have to restart the firewall or try a request you’ve not done before.

Please let me know your observations / explanations / comments / questions.

Cheers

Albert

Improvements in HTTPS Inspection Bypass mechanism - Probe Bypass

Important Note: Probe Bypass should not be used if there is a proxy between the Security Gateway and the Internet.

Limitations of HTTPS Inspection Bypass Mechanism without Probe Bypass:

• Every first connection to a site is inspected even if it should have been bypassed according to the policy.
• Non-Browser Applications connections are dropped when HTTPS Inspection is enabled (even if bypass is configured).
• Client certificate connections are dropped when HTTPS Inspection is enabled (even if bypass is configured).

Improvements introduced by Probe Bypass:

• Bypass mechanism was improved to better reflect policy and resolve the above limitations:
  • Stop the inspection of the first connection to bypassed sites.
  • Allow bypass of Non-Browser Applications connections.
  • Allow Bypass of connections to servers that require client certificate.
• New probing mechanism eliminates the need to inspect the first connection to an IP address unless it is required by the policy.

Limitations of HTTPS Inspection Bypass Mechanism with enabled Probe Bypass:

• HTTPS Inspection will not work for sites that require SNI extension in the SSL "Client hello" packet.

Status of Improved HTTPS Inspection Bypass feature (Probe Bypass) is controlled by the value of kernel parameter enhanced_ssl_inspection:

[Vladimir]

Thank you for sharing.

Unfortunately, in R80.10 resource tracking via proxy is broken and I have not seen any remediations listed on R80.20 EA features list.

[Albert_Wilkes]

Thanks for your input, I was now able to test this on R80.10 and the same situation seems to have applied. Can you elaborate what you mean by resource tracking as the tests seem to have successfully blocked traffic according to the policy even when working with the CP as a proxy

[Vladimir]

I am referring to logging of client connections to URLs.

Please see the post starting with "I can confirm that the proxy connections are not being logged properly." in this thread, (it's at the very bottom):

https://community.checkpoint.com/thread/5334-how-to-configure-check-point-security-gateway-as-httpht... 

[Albert_Wilkes]

Thanks for this but I found it works for me ... 

For some reason there are two different "Resources" in the "SmartLog" Display Profile, maybe the guys are using the wrong one? As you can see above one of them is indeed empty.

[Vladimir]

It’s tracking sessions to the proxy only.

Can you tell from these logs if the client has succeeded in connecting to the actual destination?

Vladimir Yakovlev

973.558.2738

vlad@eversecgroup.com

[Albert_Wilkes]

As per this screenshot that I used in the topmost article it was the same in R77.30: 

AFAIK, only TCP connections to the proxy are logged with the "proxied destination" information in the resource field ...

Yes, I am able to use HTTPS inspection through the proxy as well as per the below example and I get good logging for it, even when (or rather especially when) using the CP as a proxy

... and the webpage loads with my "personal" HTTPS inspection CA as expected...

So as far as I am concerned it all works ok for me, and I am also getting the URL (or rather the hostname) in both the Description AND the Resource. Your're right in that the destination is the cluster but I am pretty sure it was the same for R77.30 and in that sense nothing has changed between the two versions or I am misunderstanding the issue.

[Vladimir]

Thank you for pointing out two "Resources". I seem to be able to do the same as you are showing in the screenshots above.

Can you tell me if you have any luck getting user data in the "Source User Name" (with IA enabled), or anything at all in the URL columns (there are two of those as well)?

What i dislike from relying on this approach is that we are not seeing a complete picture of users successfully accessing the sites and  the IPs those sites were served by.

Thank you,

Vladimir

[Albert_Wilkes]

you're right about the destination IPs, it seems that information is lost which I dislike too and it'd be a good feature request and enhance the value of the logs if the proxy IP would be replaced with the final destination's IP - or maybe they'd even have an added column in the logs even?

Potentially this also prevents GeoBlocking information from showing in the logs which obviously is done by IP. I hope to be able to test IA and/or Geoblocking in the future.

What I do want to stress is that I believe all is working at the same level/quality/features as it was in R77.30.

Given the benefits in logging and/or only bypassing those requests you actually want to bypass and not all of "cloudflaressl", "akamai" etc. when using the CP as a proxy, this tradeoff needs to be considered:

Standard CP processing HTTPS requests(without proxy):

Pro: You get destination IP information in the CP logs.

Cons: You bypassing and logging will be fuzzy as it does this based on certificate information when requests are not inspected (but just categorized or not even that). Only SSL inspected traffic is logged with correct URL information.

CP as a proxy

Con: You lose destination IP information. However, you're able to regenerate the information from the hostnames/URLs even though there could be "some" fuzziness as well: host to IPs resolutions could have changed and some hostnames resolve to multiple IPs and you thus never know which of these were actually contacted. But does it really matter?

Pro: You perform bypassing and logging based on the hostname in the CONNECT rather than on cached IP information. For destinations where one IP hosts multiple websites you can bypass one and still inspect the other. This is just not possible at all without using a proxy request.

[Albert_Wilkes]

When considering the logging and policy/bypass advantages of turning on the proxy for HTTPS I did find out about one of its limitations the hard way: You'd think that sk62700 would disable tcp timestamps or (at worst) would copy the clients use of timestamps / RFC1323. Unfortunately applying sk62700 has no effect on this but in fact timestamps are ALWAYS enabled. Some websites behave badly when using the CP as a proxy and I attribute this to the use of the timestamps.

When using the CP as a proxy the capture shows a tcp syn with timestamps and in many cases isn't responded to by a website.

When hide NATting the traffic behind the GW a syn without timestamps is passed on and always receives a timely response.

This is a well-known issue (e.g. Web Site Pages Load Slow when RFC1323 is Enabled on the ProxySG ) for other proxy admins and I found it best be disabled. Shame to see this is currently not possible on the CP.

Adding that to the fact that traffic is no longer accelerated when going through the CP proxy ( as per Performance impact from enabling HTTP/HTTPS Proxy functionality  ) it seems hard to justify my previous recommendation to use the proxy whenever possible.

[Albert_Wilkes]

The proxy SK How to configure Check Point Security Gateway as HTTP/HTTPS Proxy  has now been amended by the parameter cpas_tcp_do_rfc1323 which allows to disable RFC1323 on proxy connections (default is 1).

I was unable to resolve this through a case with CP support. However a comment that I made towards the sk62700 (which I thought should have applied to the problem) has triggered someone who understood what I was requesting to own the issue rather than to ask for evermore debugs. While it took a long time (4 weeks) until I had the solution and there was no possibility to track progress, it was effortless (seen from my side - especially when compared with the support ticket that went nowhere).

To conclude, I can highly recommend commenting on SKs. The team that reviews the comments is doing a great job.