HTTPS Inspection causing slow performance to our web servers

[Parabol]

Hi all, recently we had some issue with slow connections to various websites that we host in our environment. We failed over our perimeter firewall cluster, so the standby became primary, which seemed to temporarily restore performance. We open a ticket with TAC, provided them logs and we are waiting further analysis from them.

The issue happened again, this time we set HTTPS inspection to bypass for some of the impacted websites, and we noticed that performance was restored to normal. We will update TAC with this information. But I was wondering if there is anything else we can check in the mean time? Utilization metrics (cpu/memory) seem fine, the perimter firewall isn't overburdened. I wonder if we're hitting a throughput bottleneck perhaps? Interesting that the failover restored performance though.. so seemingly the degraded performance returns over time. Maybe as more connections accumulate? 

The cluster is running on latest R81.20 JHF.

cpview showed:

1,492M Bits/sec

138K packets/sec

97 Connections/sec

2,275 Concurrent connections

Any advice or tips for further tshooting is appreciated, thanks!

[G_W_Albrecht]

Look into the history using sk101878: CPView Utility, that may give you better insights about the load during the issue!

[Lesley]

Post here also, I have similar issue. But with video hosted on the website. Causes freezes in the video with HTTPS inspection. 

Of course we have to keep in mind HTTPS inspection causes always extra load on the gateway / impact on the inspected flow. 

[Chris_Atkinson]

Which model of gateway and JHF take is installed on it out of interest?

Also how does your SSL inspection policy structure look?

[the_rock]

I think if you sent us what https inspection policy rules look like, it would help us give you a better idea. Just blur out any sensitive date.

Best,

Andy

[Timothy_Hall]

The reason performance increased upon failover is that while connections can survive a failover via state sync, streaming inspection cannot.  Therefore after failover by default any existing HTTPS connections will no longer be actively streamed and will go fastpath on the newly-active member; this is also true of existing connections that were passively streamed.  So for as long as the browser speculatively maintains these connections post-failover, the fastpath HTTPS site performance for them will be excellent.  This strange-looking effect caused by failovers is documented in my Gateway Performance Optimization R81.20 class.   When a brand new TCP/443 connection is started on the newly-active gateway, proper active streaming inspection resumes.

When you say bad HTTPS performance, can you please clarify?  Do you mean bad performance for long-running downloads via HTTPS?  Or is HTTPS user site navigation slow: either taking a long time to start loading once a user first visits a brand-new site, or slow loading/response when navigating links on the exact same website?  The answer to these questions will help point towards where we need to look.