HTTPS Inspection Probe Bypass: To enable or not to enable?

[Aaron_Vivadelli]

I've heard of mixed results on probe bypass for HTTPS Inspection and I wanted to get feedback.  To me, it seems like this is a better way of deploying HTTPS Inspection to minimize problems with bypassing traffic from inspection, but it is not something that's enabled by default which makes it seem like it should only be used if necessary.

Does anyone know of probe bypass causing more issues than it solves?  Or vice versa?  I know one solution doesn't fit all environments, but I'm wondering if there is a recommendation one way or another.  To me, I think it's something that is better enabled.

Here is the SK:  HTTPS Inspection Enhancements in R77.30 and above 

Aaron

[Alain_Ikula]

Hi Aaron,

Probe Bypass should be enabled if the legislation in your country prevent you from inspecting some traffic (banking and financial categories mainly) using HTTPS inspection. In this case, you bypass HTTPS inspection for those categories and you enable Probe Bypass to avoid that the first packet of the connection is still HTTPS inspected despite your policy. And how is the HTTPS site categorized without HTTPS inspection and with Probe Bypass enabled ? Well, you may use the 'categorize HTTPS websites' functionality but there are limitations (sites using Wilcard certificates, UserCheck). See sk92743 for more details.

Regards,

Alain

[Aaron_Vivadelli]

Thanks for responding Alain.

I work for a US based VAR and perform many integrations for our customers of all CP products.  All of my customers are sensitive about decrypting financial and healthcare related sites, which is why I feel probe bypass is a good thing to implement at all my customers.  I wanted to get feedback from others though because I heard varying recommendations surrounding if this feature causes more problems than benefits.  I think there have been a number of fixes in recent JHFs though (at least for R77.30).

Usually I like to deploy HTTPS Inspection with a gradual approach (even recommended in the HTTPS Inspection Best Practices SK), but bypassing rules are theoretically ineffective without Probe Bypass enabled since the first packet is still decrypted.

I guess I'm looking to see if anyone has had specific issues with Probe Bypass enabled where they were better off turning it off.

Aaron

[Matt_Snead]

I've tried it a couple times and always end up disabling it because there are too many incompatibilities.  The biggest one is outlined in sk104717:

Limitations of HTTPS Inspection Bypass Mechanism with enabled Probe Bypass:

• HTTPS Inspection will not work for sites that require SNI extension in the SSL "Client hello" packet.

It really is a shame because I too do not want to be inspecting people's traffic.  I only want to prevent them from going to specific domains.  So far this seems to be impossible with checkpoint as https categorization is also flawed.

[PhoneBoy]

You may want to use the App Control Signature Tool in this case.

Signature Tool for custom Application Control and URL Filtering applications 

[Albert_Wilkes]

Can you explain where this would help or which of Matt's statements your suggestion refers to (categorization, probe bypass?)? Are you saying you can (or have to) use the App Control Signature Tool to create a (non-IP based) pattern to bypass websites that require SNI even if probe bypass is enabled? My customer found that he can only do a bypass by IP which is also mentioned in sk112066 which is why I am evaluating to disable probe bypass and use the CP as a proxy which should improve things as per the bottom section of sk92888

[PhoneBoy]

Basically what you're doing is creating an App Control Signature that is looking for the SNI.

It does not solve the Probe Bypass issue, but it does give you a way to allow access to specific websites without being required to enable HTTPS Inspection.

[Matt_Snead]

I’m confused... probe bypass is an enhancement to https inspection. What do you mean by, “without being required to enable https inspection?”  Are you referring to people who don’t use https inspection at all?  In which case your solution is simply one to allow a blocked https site through?  E.g. nothing to do with https inspection or probe bypass, per say?  Pretty sure that’s not what this thread was about. 

[PhoneBoy]

My comment was in response to this statement you made above:

It really is a shame because I too do not want to be inspecting people's traffic.  I only want to prevent them from going to specific domains. 

If you don't want to be inspecting people's traffic (i.e. enable HTTPS Inspection) but want a way to prevent people from going to specific domains...you can create a custom signature for those domains.

This does not require enabling HTTPS Inspection at all (and thus not needing probe bypass).

I hope that clarifies things.

[Olga_Kuts]

Сan someone explain me the principle of Probe Bypass work?

Thanks.

[PhoneBoy]

Limitations of HTTPS Inspection Bypass Mechanism without Probe Bypass:

• Every first connection to a site is inspected even if it should have been bypassed according to the policy.
• Non-Browser Applications connections are dropped when HTTPS Inspection is enabled (even if bypass is configured).
• Client certificate connections are dropped when HTTPS Inspection is enabled (even if bypass is configured).

Improvements introduced by Probe Bypass:

• Bypass mechanism was improved to better reflect policy and resolve the above limitations:
  
  • Stop the inspection of the first connection to bypassed sites.
  • Allow bypass of Non-Browser Applications connections.
  • Allow Bypass of connections to servers that require client certificate.
• New probing mechanism eliminates the need to inspect the first connection to an IP address unless it is required by the policy.

Limitations of HTTPS Inspection Bypass Mechanism with enabled Probe Bypass:

• HTTPS Inspection will not work for sites that require SNI extension in the SSL "Client hello" packet.

HTTPS Inspection Enhancements in R77.30 and above 

[Olga_Kuts]

Thanks for reply. I understand limitations and improvements, but I want to understand the mechanism of probe bypass work. How is the bypass done if the connection is not inspected?

[PhoneBoy]

Read the article Albert Wilkes‌ linked above, it shows what happens when enabled.

[Albert_Wilkes]

Please read my update below. It starts a probe / ssl negotiation to the destination before it even responds to the client with a SYN/ACK. Let me know in case there are any questions after reading the below and linked article

[Olga_Kuts]

Great work! Thanks!

[Albert_Wilkes]

my article might a bit of insight into the technicalities of probe bypass. In short probe bypass opens a connection to the server based on the destination even before taking the decision on whether to bypass the inspection or not. A single SYN request on the client to fw side is sufficient to start an SSL negotiation on the fw to server side as can be seen in the case where it breaks when SNI is required

HTTPS inspection real life examples and caveats in R77.30 

[John_Fenoughty]

Following Arron's question and having read and re-read Albert Wilkes's excellent document. I had both the requirement and inclination to carry out testing on my own test systems and then over the last four months on various live sites.

The attached 10 page document contains my findings and the relevant technicalities. The introduction and summary are clear: in almost all cases, disabling probe bypass made things a LOT better! That said, there are factors that can obfuscate this universally (in my experiences) positive result until they are addressed.

My document uses various test sites (13 in the first instance) and one of those, Skype gets a good four or five pages of the document. Summary: I now have Skype working well on a number of sites with HTTPS enabled on those computers using URL based bypasses only (no IP address or ranges).

I will also be posting this in the main thread about getting Skype to work with HTTPS inspection enabled, these two things being so inextricably linked I could not separate them in to two documents.

My document and the CSV file with the URLs for Skype are available in the 'Documents section of 'GeneralProductTopics' - The document is named 'Making Skype work properly with HTTPS inspection enabled coupled with To Probe Bypass or Not To Probe Bypass 1oct18' and the current CSV is SkypeURLs Version 2.3 August 2018.csv.

[Matt_Snead]

Great post, thanks for sharing.  Checkpoint really needs to get their S together and keep on top of changing technologies if they want to keep customers.  Their security practices are top-notch, but their general filtering/inspection policies are way behind.  Couple that with the many issues with emerging SSL ciphers/elliptical curves and it's becoming an common nuisance.

[PhoneBoy]

Thanks for putting this together!

It'd be great if you could upload this as a document to General Product Topics‌, which will make it easier to find.

[Marcel_Wildenbe]

Hi John,

Thanks for the extensive explanation of the Probe Bypass feature. I have created the Application/Site Group with all the Skype related objects, but when I try to use this object in the https inspection policy under "Site Category", I get the following error:

"HTTPS Inspection: rule 1. In 'Site Category' column,  applications or groups with applications are not supported."

Any idea?

[John_Fenoughty]

Ah, yes - when I re-read my document I can see that I need to make more of the transition from the URL an Application filtering section to the HTTPS inspection policy section.

Anyway, the group with all the various objects for the applications is to be used in the URL and Application filtering policy in the R80.x 'unified' policy but in the R77.30 style HTTPS inspection policy the bypass needs to be just a single Custom Application with URLs.

I hope that clears things up. - and that you find success with this process. I'd be very pleased to hear if you do.

[Marcel_Wildenbe]

"I hope that clears things up" Well, actually, it doesn't 😉 We are running R80.10 (Man + GWs) and I have created a rule in the https inspection rulebase just like in the screenshot you've used.

[John_Fenoughty]

Marcel,

You are absolutely right - I have just checked the situation on a real manager (rather than the Demo) ! I recycled a group for the HTTPS rulebase screenshot and it is in error!

The HTTPS rulebase, which I called 'R77.30' because even when you use R80.10 the HTTPS rulebase is still in the 'R77.30 dashboard 'style' - indeed a Smart Dashboard is spawned when you click on the HTTPS Policy link in R80.10.

In the HTTPS policy you must use a single 'Custom application' with the CSV of URLs in it. I make this clear in words on the bottom of page 8 but you are correct that the screenshot at the top of page 7 has a group in it and this (currently) will not work- until Check Point move the HTTPS policy properly in to the Unified Rulebase (the R80.10 style). I made this error as I was reproducing the screenshots in the Demo Smart Console so I did not install that policy with the error in it :-}

I am going to fix the screenshot and put the correct wording on page 7 where the error is. Thank you for pointing this out.

I'll be using this screenshot to show the correct rule for the override in the HTTPS policy.

Don't forget all the STUN ports and so on, which need to be in the main access policy!

Good luck, please let me know how you get on - and if I have made any further errors in the document!

Version 4 here we go...

[Marcel_Wildenbe]

Update: I have configured the https inspection rulebase according the latest suggestions and presto... it works.

I can have a conversation with Skype4Bussines within our domain and Skype4Business and Skype4Consumer outside our domain. Chat, video, sound and status information is conveyed flawlessly. Only sharing of documents fails, but this might have to do with setting for Skype within our company.

In the logging, I can see the bypass rules being effective.

In a Skype4B to Skype4C conversation, I see a lot of dropped traffic from the Skype4B client towards an IP-address within the IP space of the ISP of the Skype4C client (UDP/14472 and UDP/32442). It doesn't effect the conversation as far as I can see.

[John_Fenoughty]

That's great to hear - thanks again for the correction to my document, there's nothing like a person actually using it to properly debug a document!

I have observed that file transfer between Skype4B and Skype4C  (either direction) fails even when both computers are directly connected to the Internet. Likewise Screen Sharing between Skype4B and Skype4C is a no-go, the option is greyed out or disabled.

I have to admit I can no longer recall if file sharing between Skype4B and Skype4B works behind the Check Point with HTTPS enabled. TBH, if it did, I'd consider blocking it with the content blade anyway!

[Darran_Lebas]

Hi John,

I've read your 'Making Skype work properly with HTTPS inspection' guide.  I can see that you are in fact referring to using the default out of the box configuration and not enabling the enhanced probe bypass as being the most reliable solution.

This causes me a dilemma now as I was hoping probe bypass was the answer to all of my issues (excluding the SNI problems that it brings). I have problems with a range of applications such as Whatsapp, similar messaging apps, Skype and others. I also have issues bypassing HTTPS inspection on domains names. TAC have told me in the past that this can't reliably be achieved without enabling Probe bypass. Is this not the case? I'm wondering if the bypass on the domain is being applied but the application is failing due to the first packet is being inspected?

Currently, as it stands, I really am struggling to provide useable solutions to my organisation with HTTPS inspection enabled. Productivity is at risk to thousands of users who expect things to work and I have no easy way of fixing this. This is compounded by the fact that I'm running VSX and I can't enable probe bypass on one virtual gateway as it's a global setting across all VSX GW's.

Checkpoint, I'd love to hear your thoughts on this as I'm sure I'm not the only person struggling here. 

John, you mention in your guide that Skype wouldn't work with PB enabled due to some of the URLs being hosted on content delivery networks and using SNI. If we were bypassing the Skype URLs, wouldn't it work anyway?

[John_Fenoughty]

I didn't get the chance to reply before you answered it for yourself, that yes my findings are that Probe Bypass does not help with technical issues and brings many problems with the HUGE number of sites and services requiring SNI to work.

I don't think I've allowed WhatsApp on any networks with HTTPS inspection enabled as it's mostly for smartphones and my personal approach to those is (with a very few exceptions) to have a separate 'insecure' zone or 'dirtynet' for them away from the corporate network. I'll have a play with it in my lab and see if I can make it work with HTTPS inspection enabled. At a guess, I reckon it will need and override because it is encrypted client server application and so *should* be using a sticky certificate at the very least if not a full public/private key certification to defend itself against man in the middle attacks. We'll see...

I do know that Check Point have continued to put in a lot of work on this at every Jumbo hotfix and indeed thatere has been a lot more done in R80.20 which was recently released. You have not mentioned in your post the version and jumbo level you are running at your gateway(s). If you are not on the very latest Jumbo I would absolutely say that is your first priority and then if you are able to go to R80.10 at least and R80.20 if possible then that would be the next move.

I am yet to deploy an R80.20 in a live environment but it won;t be very long.

To your last point, no the SNI problem affects the very initial connection will break an SNI based site even if you attempt a bypass by FQDN or wildcard it make no difference, it's that catch-22 style issue - it needs to inspect the initial connection to see the FQDN to then see if that has an exception by which time it is too late and the conneciton is 'broken'. Check Point have stated that they will not just 'trust' the URL requested in the as this has security implications (and it most assuredly does) but they have previously said that they might be working on extracting the URL from the certificate and including a way to make this reliable and secure. There's always the wildcard certificate problem to make that more complex as well

Check Point continue to evolve this technology and they really are getting better at every turn, but it's still not 100% perfect.

Ultimately in my opinion while Check Point have achieved an excellent solution under difficult global circumstances, total perfection is not achievable with the Internet as it is and that the currently mooted possibility of HTML6 having an API in to which a corporate gateway can connect for in-stream decryption purposes. You can imagine how divided the internet community is on that one!

[Darran_Lebas]

Thanks for the reply. I've cancelled my planned changes to implement PB next week now.

I'll give your Skype tweaks a try next week. If I can get this working it would really take the pressure off.

Reference WhatsApp, ideally we would be routing this traffic to another firewall but currently, this isn't the case.

My firewalls are bang up to date and using the latest just released JHF.

Yeah, I know it's not an easy thing to implement but when you're constantly being asked 'why doesn't this work', and 'why can't we use this', you do quickly fall out of love with HTTPS inspection.

[Albert_Wilkes]

John Fenoughty wrote:

...

Ultimately in my opinion while Check Point have achieved an excellent solution under difficult global circumstances, total perfection is not achievable with the Internet as it is.

Absolutely agreed!

[Timothy_Hall]

Agreed as well, and when students bring up the numerous issues they have encountered with HTTPS Inspection in class my opinion is always this:  We are approaching the limits of what a network-based device like a gateway can do sitting in the middle of the network between clients and servers trying to inspect encrypted streams of traffic.  Browsers are getting smarter and smarter about detecting any kind of man in the middle attempt, which of course is the browser trying to increase user security, but interferes with the ability of the "good guys" like us to help protect users utilizing HTTPS.  All firewall vendors are experiencing issues with HTTPS Inspection, not just Check Point.

I'm a network guy through and through, but even I can see that it sure would be easier to just inspect the data right on the user's workstation before it is encrypted by the user's browser in the first place.  Check Point definitely has agents to do this, and I think Dorit Dor‌ has talked about a future of "nano agents" everywhere to help secure endpoints and detect and block things like ransomware attacks.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com