topic Re: R81 HTTPS Inspection in General Topics

R81 HTTPS Inspection

gtsanava — Fri, 24 Sep 2021 07:33:22 GMT

Hello

we have upgraded from R80.30 to R81 and we discovered that HTTPS Inspection has some problems there.

for example on youtube.com some video images are unable to load, these images are loaded from ytimg.com. after bypassing ytimg.com issue is resolved.

same repeats not only for youtube.com but for several websites, for example: workspace.com, admin.exchange.microsoft.com, ui.com, for some facebook.com, microsoft.com pages and many more.

In overall it looks like some content can't be inspected and is stuck somewhere.

I can't just bypass everything :))) maybe there is some fix or any experience about it? you can see my settings in attachments.

Respectfully,

George Tsanava

Re: R81 HTTPS Inspection

_Val_ — Fri, 24 Sep 2021 08:20:36 GMT

Do you actually have HTTPS enabled? How does your inspection rulebase look? Any errors/weird entries in the HTTPSi logs? Is your GW capable to resolve DNS for external servers, so SNI verification would work?

Re: R81 HTTPS Inspection

gtsanava — Fri, 24 Sep 2021 08:58:28 GMT

if you mean HTTPS Inspection then yes it is enabled, I have uploaded my inspection rule base screen too, no there are not any weird entries in logs, yes gateway can resolve any servers, it has access to internet.

Re: R81 HTTPS Inspection

_Val_ — Fri, 24 Sep 2021 11:34:31 GMT

Thanks. Rule 5 does not look good. Put "Internet" instead of "Any" as Destination. Also, Source field is also questionable, I would advise using "internal networks" instead of Identity Awareness user role.

How does your cleanup rule look?

Anything related to performance bottleneck? CPU utilization on particular cores?

Re: R81 HTTPS Inspection

_Val_ — Fri, 24 Sep 2021 11:40:21 GMT

You may also want to look into sk163595 & sk112214

Re: R81 HTTPS Inspection

the_rock — Fri, 24 Sep 2021 13:10:35 GMT

I had case with TAC escalations for https inspection for few months and we actually found that kernel parameter mux_enabled was causing some issues, so one thing you can try is this...if you have a cluster, run below command on whichever member is active:

fw ctl set int mux_enabled 0

Then push policy and test again. If this fixes your problem, not sure I would recommend leaving it that way, so you may want to open TAC case and have them run a debug when feature is set to 1, which is default. It supposedly has something to do with streaming and how traffic is distributed, but also plays big part on inspection as well.

I pasted below link about it:

MUX module

Re: R81 HTTPS Inspection

the_rock — Fri, 24 Sep 2021 13:14:20 GMT

Personally and this is just my opinion, yes, @_Val_ is correct about rule 5, but I am 99% sure it wont make any difference even if you change it. TAC and myself tried playing around with rules dozens of times when customer had similar issues and it did not change the behavior at all.

Re: R81 HTTPS Inspection

_Val_ — Fri, 24 Sep 2021 13:59:16 GMT

Rule 5 in the current state tries decrypting all the traffic passing through the GW. It may not make a difference, but definitely needs to be changed to the best practice. It many cases it saves you a lot of CPU%

Re: R81 HTTPS Inspection

the_rock — Fri, 24 Sep 2021 14:02:29 GMT

Again, I agree with you, but just speaking from my own experience with TAC :). Tried it many times, did not change cpu, memory usage or any other behavior at all.

Re: R81 HTTPS Inspection

_Val_ — Fri, 24 Sep 2021 14:14:25 GMT

Sorry to say, I cannot find a single case with "fw ctl set int mux_enabled 0" recommendations, related to R81. With all due respect, @the_rock , I believe the topic starter should follow TAC recommendations instead of playing with kernel parameters.

@gtsanava please open a support case.

Re: R81 HTTPS Inspection

the_rock — Fri, 24 Sep 2021 14:21:13 GMT

I actually have maintenance window with a customer tomorrow where we will have remote with TAC, so they can take debugs for 2 really odd issues when that kernel option is set to 1. Will see how far we get...

Re: R81 HTTPS Inspection

Ilya_Yusupov — Fri, 24 Sep 2021 14:29:36 GMT

Hi @gtsanava ,

Since R80.40 we supporting HTTP2 inspection, which means in R81 you are passing through http2.

Please open tac case and share the number and i will push it from my sids.

Thanks,

Ilya

Re: R81 HTTPS Inspection

_Val_ — Fri, 24 Sep 2021 15:33:55 GMT

Just saying, playing with MZX without TAC recommendations is not a good idea.

Re: R81 HTTPS Inspection

the_rock — Fri, 24 Sep 2021 15:36:20 GMT

Agree 100% and we never had...TAC escalations recommended it actually.

Re: R81 HTTPS Inspection

_Val_ — Fri, 24 Sep 2021 15:42:11 GMT

Even if TAC recommends something in your case, it does not mean any recommendations you received would work, let alone be safe for others.

You can recommend troubleshooting steps and analysis techniques, also with caution. Changing kernel parameters, or any other deep engine settings - no.

Let TAC be liable to any impact those changes cause 🙂

Re: R81 HTTPS Inspection

the_rock — Fri, 24 Sep 2021 15:44:30 GMT

Well, its always sort of catch 22 situation as they say...we rely on TAC recommendations in complicated cases like that. Anyway, but I get what you are saying 🙂

Cheers and have a nice weekend!

Re: R81 HTTPS Inspection

gtsanava — Sun, 26 Sep 2021 09:34:32 GMT

rule 5 is cleanup rule and is ok because we are using that gateway as forwarding proxy server there goes traffic only from users browsers to internet so it's completely okay.

in R80.30 I even had there any source and any destination so I put there "all identified users" after upgrade to R81 but it makes no sense.

Re: R81 HTTPS Inspection

gtsanava — Sun, 26 Sep 2021 09:37:38 GMT

I have opened service request and now waiting for remote session in 30 minutes so I 'll update you after, many thanks for experience you shared guys.

Re: R81 HTTPS Inspection

the_rock — Sun, 26 Sep 2021 11:50:57 GMT

I agree with you 100%. I can also tell you that from my personal experience, it also makes no difference what you put in there. I cant speak for other people, but having spent extended time troubleshooting https inspection with TAC escalations, that is what I experienced.

Re: R81 HTTPS Inspection

the_rock — Sun, 26 Sep 2021 11:47:27 GMT

Yes, please keep us posted, Im curious to see what the outcome will be...