R81 HTTPS Inspection

[gtsanava]

Hello

we have upgraded from R80.30 to R81 and we discovered that HTTPS Inspection has some problems there.

for example on youtube.com some video images are unable to load, these images are loaded from ytimg.com. after bypassing ytimg.com issue is resolved.

same repeats not only for youtube.com but for several websites, for example: workspace.com, admin.exchange.microsoft.com, ui.com, for some facebook.com, microsoft.com pages and many more.

In overall it looks like some content can't be inspected and is stuck somewhere.

I can't just bypass everything :))) maybe there is some fix or any experience about it? you can see my settings in attachments.

Respectfully,

George Tsanava

[_Val_]

Do you actually have HTTPS enabled? How does your inspection rulebase look? Any errors/weird entries in the HTTPSi logs? Is your GW capable to resolve DNS for external servers, so SNI verification would work?

[gtsanava]

if you mean HTTPS Inspection then yes it is enabled, I have uploaded my inspection rule base screen too, no there are not any weird entries in logs, yes gateway can resolve any servers, it has access to internet.

[_Val_]

Thanks. Rule 5 does not look good. Put "Internet" instead of "Any" as Destination. Also, Source field is also questionable, I would advise using "internal networks" instead of Identity Awareness user role.

How does your cleanup rule look? 

Anything related to performance bottleneck? CPU utilization on particular cores?

[_Val_]

You may also want to look into sk163595 & sk112214

[the_rock]

Personally and this is just my opinion, yes, @_Val_ is correct about rule 5, but I am 99% sure it wont make any difference even if you change it. TAC and myself tried playing around with rules dozens of times when customer had similar issues and it did not change the behavior at all.

[_Val_]

Rule 5 in the current state tries decrypting all the traffic passing through the GW. It may not make a difference, but definitely needs to be changed to the best practice. It many cases it saves you a lot of CPU%

[the_rock]

Again, I agree with you, but just speaking from my own experience with TAC :). Tried it many times, did not change cpu, memory usage or any other behavior at all.

[gtsanava]

rule 5 is cleanup rule and is ok because we are using that gateway as forwarding proxy server there goes traffic only from users browsers to internet so it's completely okay.

in R80.30 I even had there any source and any destination so I put there "all identified users" after upgrade to R81 but it makes no sense.

[the_rock]

I agree with you 100%. I can also tell you that from my personal experience, it also makes no difference what you put in there. I cant speak for other people, but having spent extended time troubleshooting https inspection with TAC escalations, that is what I experienced.

[_Val_]

We did discuss best practices for HTTPSi in this forum a dozen of times or more. Plese look it up. If rule 5 is the last rule, you are decrypting, or trying to decrypt a lot. Also, HTTPSi happens way before the regular rulebase is matched, so user roles might be an issue there. All inspect rules for outbound should have Internet as Destination and relevant Web services as Service. A final clean-up rule should say Any-Any-Any-Bypass

[the_rock]

Val, just for my own reference and sorry, not trying to be pest about this, just want to make sure I get all the FACTS correct. Is there anywhere on CP side officially documented what you stated in your response, because reading below link, I could not find it.

HTTPS inspection best practices 

And also, as I stated few times already, based on my own personal experience and speaking with 3 TAC escalations people, not one of them said that it really mattered at all the order of the rules or what was in the source and the reason why I believe it is because we tested it on customer's live environment and it absolutely made no difference whatsoever.

[Pawel_Szetela]

Hi Val,

"A final clean-up rule should say Any-Any-Any-Bypass" or should it contain services HTTPS default services?

[_Val_]

Any for services is a better choice.

[Timothy_Hall]

Are you sure?  The following presentation shows populated services in the final bypass rule.  Can we get a clarification on this from R&D?

HTTPS Inspection Best Practices

Edit: Please have R&D also look at my post here for accuracy: https://community.checkpoint.com/t5/Management/HTTPS-Inspection-Policy-Rule-Order/m-p/128681#M27952

[the_rock]

Im with you on that...though I cant access the link, it would be actually nice to get a consistent/official CP recommendation on this, because so far, at least me personally, I cant seem to find one.

[the_rock]

I had case with TAC escalations for https inspection for few months and we actually found that kernel parameter mux_enabled was causing some issues, so one thing you can try is this...if you have a cluster, run below command on whichever member is active:

fw ctl set int mux_enabled 0

Then push policy and test again. If this fixes your problem, not sure I would recommend leaving it that way, so you may want to open TAC case and have them run a debug when feature is set to 1, which is default. It supposedly has something to do with streaming and how traffic is distributed, but also plays big part on inspection as well.

I pasted below link about it:

MUX module 

[_Val_]

Sorry to say, I cannot find a single case with "fw ctl set int mux_enabled 0" recommendations, related to R81. With all due respect, @the_rock , I believe the topic starter should follow TAC recommendations instead of playing with kernel parameters.

@gtsanava please open a support case.

[the_rock]

I actually have maintenance window with a customer tomorrow where we will have remote with TAC, so they can take debugs for 2 really odd issues when that kernel option is set to 1. Will see how far we get...

[_Val_]

Just saying, playing with MZX without TAC recommendations is not a good idea.

[the_rock]

Agree 100% and we never had...TAC escalations recommended it actually.

[_Val_]

Even if TAC recommends something in your case, it does not mean any recommendations you received would work, let alone be safe for others. 

You can recommend troubleshooting steps and analysis techniques, also with caution. Changing kernel parameters, or any other deep engine settings - no.

Let TAC be liable to any impact those changes cause 🙂

[the_rock]

Well, its always sort of catch 22 situation as they say...we rely on TAC recommendations in complicated cases like that. Anyway, but I get what you are saying 🙂

Cheers and have a nice weekend!

[gtsanava]

I have opened service request and now waiting for remote session in 30 minutes so I 'll update you after, many thanks for experience you shared guys.

[the_rock]

Yes, please keep us posted, Im curious to see what the outcome will be...

[gtsanava]

Hello guys @the_rock @_Val_ @Ilya_Yusupov 

sorry for late response.

problem is resolved after we disabled new inspection for HTTP2 and instead use regular HTTPS inspection.

[Ilya_Yusupov]

Hi @gtsanava ,

If you are not running JHF 42 and above so if it's possible for you to try latest take?

Asking since we have major fixes around HTTP2 that should address  to such cases.

In case it's not solving the issue please ping me and  i will engage RnD for further investigation.

Thanks,

Ilya  

[gtsanava]

I'm running JHF 36 and still didn't received JHF 42 .

we also had problem with RA VPN which is resolved from JHF44 so I think that we will skip JHF 42 and we will go for JHF 44, or if JHF42 will not cause same RA VPN problems we will not skip it 🙂

Thanks

[Ilya_Yusupov]

Go ahead with JHF 44, in any case if you have any problem feel free to contact me offline and i will do my best to assist.

Thanks,

Ilya 

[Ilya_Yusupov]

Hi @gtsanava ,

Since R80.40 we supporting HTTP2 inspection, which means in R81 you are passing through http2.

Please open tac case and share the number and i will push it from my sids.

 Thanks,

Ilya