Apple services not working with HTTPS inspection

[Austin_Ponten]

Hi, First post so I will do my best 🙂

The environment is R80.40 Take196 

The issue:

We are trying to perform HTTPS Inspection for our trusted client networks for a customer. The problem arises when Apple per their post here https://support.apple.com/en-us/HT210060 state "Attempts to perform content inspection on encrypted communications between Apple devices and services will result in a dropped connection to preserve platform security and user privacy."

So, we created exceptions for this in the policy. I followed the guidelines listed in these:

https://community.checkpoint.com/t5/Security-Gateways/Apple-and-HTTPS-Inspection/m-p/176059/highligh...

https://support.checkpoint.com/results/sk/sk108191

https://support.checkpoint.com/results/sk/sk112994

The First two links did not help, as the redirects via AKAMAI Tech did not get caught by the Bypass exceptions no matter how many *apple.com domains or certificates were added. 

The last link where in step 3 it states to:

"Create a Network object that specifies the relevant AKAMAI network (based on the example above - 88.221.0.0/16)"

Does in fact make the exception for inspection work, but my client nor I find this as a valid solution as Apple is not the only tenant for AKAMAI Tech. 

The question I present to the community is: How can I perform content inspection on ONLY Apple-related traffic WITHOUT compromising my internal client networks?

I can provide additional information if needed, and thanks for reading my first post 🙂 

-A

[G_W_Albrecht]

I woud suggest to contact TAC to get an official solution !

[Austin_Ponten]

Hi thanks for the quick reply. Yes, I initially went to TAC to get help and after a bit of testing and labbing gave me the following answer "Unfortunately, without having the proper ranges for AKAMAI, it is trickier to find a way to bypass only the specific services without the knowledge of the specific IP addresses for AKAMAI." and closed the case. 

-A

[G_W_Albrecht]

So all is clear - official solution is to use IPs / IP Ranges.

[Sorin_Gogean]

Hello @Austin_Ponten ,

Can you be more specific on the problem you're seeing?

I was setting up recently several Apple Cache servers and you can see below the firewall rules we have for those boxes. 

(we have HTTPS Inspection enabled for ALL HTTPS 443 traffic)

The custom objects are below (no REGEXP):

apple.com object	c.apple.news object	App Store Object
*.apple.com .apple.com .icloud.com *.icloud.com appleid.cdn-apple.com .cdn-apple.com *.cdn-apple.com	c.apple.news .apple.news *.apple.news	apps.mzstatic.com *.mzstatic.com .mzstatic.com .icloud-content.com *.icloud-content.com

With all those settings, I can tell you that I see the Apple Cache machines, being able to communicate with Apple Cloud, and the packages are downloaded/validated without any issues. 

Thank you,

[Austin_Ponten]

Hi, Thanks for your quick reply. 

I followed your suggestions to a T as a test for our implementation, but for us, our Mac users cannot initiate a software update, run iTunes, download or search apps in the store etc. (all Apple-related user content).

The thing is, that out of 10 connection attempts, I would say maybe 1-2 will actually work as the traffic is not redirected via AKAMAI and the user is able to reach iTunes or the App Store using the exceptions I had made either from your suggestion or a combination of 17.0.0.0/8, domains, and certificate additions. 

The problem is slightly different I guess than those Apple cache machines I'm afraid.. 

-A

[the_rock]

I worked for some time with customer who has 95% of their environment Mac books and Mac minis, so Im fairly familiar with this. Sadly, I dont have access to their environment currently, so will see if I can find some screenshots/notes about how we made this work. I do recall we added whatever we could find when searching Apple in updatable objects for https inspection bypass, and then also added all Apple IP ranges to be allowed as well.

[Sorin_Gogean]

Hello @Austin_Ponten ,

I don't remember any AKAMAI addressing while looking for the Apple traffic permissions .

The example I give you, with Apple Cache, as you addressed issues with Apple traffic, and Apple Cache is ONLY Apple traffic - packages and updates .

On those failure, can you show some logs - I really don't get it why ppl are so shy in showing logs 😕 - as those can point us other to other ideas.

Thank you,

[Austin_Ponten]

Hi, It is more out of laziness that I wasn't showing any logs as I don't have access to a test Mac user readily. I did procure a history log from last week's testing that displays the problem at hand. 

It shows the user attempting to reach Software updates from Apple, but the destination is akamaitechnologies.com which gets inspected since it is not defined in the Bypass exceptions for Apple and then it is dropped.

-A

[the_rock]

Just create custom app category and add *akamai* and make sure its allowed in regular layer, as well as https inspection policy and test.

Andy

[Austin_Ponten]

That might work yes, but the whole point that I am trying to do is to avoid this "any any" rule, As far as I am aware Apple is not the only company that uses Akamai as a CDN and I would like to not create a larger exception than needed. 

I don't know what specific customers they have, but my first Google result is https://www.appsruntheworld.com/customers-database/products/view/akamai-cdn

-A

[Sorin_Gogean]

I see, so your issue was not with the firewall access permissions, but the HTTPS Inspection part.

On our side we're having this rule - see below - and on the Custom Applications we're setting a 2nd category "HTTPS_Inspection_Bypass" that is set here not to be inspected.

You DO NOT NEED to except AKAMAI as you think, since all calls are done towards apple.com or similar domains, like I showed you in the previous screenshots/tabels .

Thank you, 

[Austin_Ponten]

Interesting. Yes I have the Custom App object for Apple, but what is in the custom category HTTPS_Inspection_Bypass that you use to catch Apple-related traffic that I might be missing?

-A

[Sorin_Gogean]

As stated previously "on the Custom Applications we're setting a 2nd category "HTTPS_Inspection_Bypass" that is set here not to be inspected" -  it's an EXTRA Category that we set on Custom Objects - the ones I showed in the table, earlier .

Ty,

[the_rock · Accepted solution]

K, found it from my notes...so here is what we bypassed and worked fine.

Andy

 Sadly, you cant add much from updatable objects, as this is only thing that pops up and in all honestly, its utterly useless.

 

[G_W_Albrecht]

Not useless, but only usable on SMB appliances ! Akamai has an Updateable Object...

[the_rock]

I had client tell me before they had TAC add it for them for SMB as well and it did absolutely nothing. Anyway, you are correct, Akamai updatable object is there, so that may help, for sure.

[G_W_Albrecht]

Yes - as it currently does only work with locally managed SMBs it does absolutely nothing in Dashboard 😎

But as we read in sk131852: Updatable Objects this is an object in Checkpoint Cloud, not in Dashboard or SMB WebGUI - and that is the reason that it is listed, can be selected in Dashboard and SMB rules, but will only work on locally managed SMBs.

[the_rock]

Well, that explains it then 🤣. Customer was using centrally managed SMB and he said TAC told them was fine with that object, regardless locally or centrally managed...go figure lol

[G_W_Albrecht]

It is all in the SK:

Quantum Spark Smart Accel

Improves connectivity and optimizes the load on the Quantum Spark Security Gateway. Once enabled, traffic enforcement is accelerated for selected services. Note: Firewall and logging activity is not affected. Smart Accel is currently in EA and is only supported in locally managed Gaia Embedded appliances / Quantum Spark Security Gateways running version R81.10 and higher.

[the_rock]

Yes sir, I agree 100%. Customer even pointed this out to TAC, but they still insisted it was supported on centrally managed SMB...where they got that info, I wont even dare to make a guess 😎

[G_W_Albrecht]

Mistakes are a common part of our lives 😎 When this feature was presented, it took some time to learn that it is still limited to locally managed Gaia Embedded appliances, but intended for broader use...

[the_rock]

That is true 🙂

[Austin_Ponten]

I have done a variant of this, including the custom objects that @Sorin_Gogean had suggested, but not the exact looking one as yours, I will add this to the policy and see if there is any change. You only had this destination custom app object in the rule then?

-A

[the_rock]

Yep and I forgot *icloud*, it was there as well.

Andy

[the_rock]

I would take advice from @Sorin_Gogean . He is, in my opinion, the BEST when it comes to https inspection. I say that, since he helped me big time for an issue I had last year and the way he explained things was superb 💪

Andy

[Austin_Ponten]

I tried this and removed all my other bypass rules that I was testing with and such, and the first test worked! I won't call the Nobel org just yet, but if this does end up lasting the next 24 hours I will almost have more questions than answers...

Thanks for the input!

-A

[the_rock]

Glad to hear. BUT, here is what I will tell you...with customer I was talking about, we made exact same rules in ordered layer (where fw and url/app[ control blades were enabled, as they came from Cisco, so did not feel comfortable with multiple ordered layers) and https inspection policy, meaning we bypassed that custom object with 17.0.0.0/8 range, *apple*, *itunes* and *icloud* and all worked fine. I dont have access to their environment, as they feel comfortable fixing their own CP issues now and dealing with TAC if needed, but Im happy to take screenshots in my lab to demonstrate, as it would be very similar to how they have it.

Let me know if thats needed, but if not, keep us posted.

Andy

[Austin_Ponten]

Hi, So far its been looking better since this addition to the HTTPS policy. No complaints from Mac users since yesterday.

But, I found another issue when I tried to apply this same bypass rule to an HTTPS policy on a different FW with the same customer and (as far as I can tell) with the same settings. Maybe @Sorin_Gogean might also know what this means?

I'm reading https://community.checkpoint.com/t5/General-Topics/HTTPS-Inspection-Probe-Bypass-To-enable-or-not-to... and it seems relevant although I am not running older versions and I even checked my kernel parameters to ensure that it was set to default. 

Is there something I'm missing? It makes sense in a way that if I probe Apple and they reject that I can inspect the traffic, the connection is then dropped. But I don't remember having to set any values or settings on the other FW that seems to be working now.. 

-A

[G_W_Albrecht]

Found a similar error here: Trouble with the MACs after hardening Cypers/TLS/SSL

This also seems interesting sk180807: macOS does not connect to the on-premises Endpoint server with FQDN