Delay when visiting sites hosted at Cloudflare

[JasMan]

Hi all,

I noticed that some of the websites that I visit daily need a very long time until the first content shows up in the browser. I run a tcpdump on my client, the LAN and the WAN site of our perimeter firewall to analyze the cause of the delay.

The SYN packet from my client hits the LAN interface of the firewall after 1-3ms. But the outgoing SYN on the WAN site of our firewall appears 5-6 seconds later.
The logs in the SmartConsole draws a different picture: the incoming and outgoing SYN appear in the log at the same second, which is exactly the time where the outgoing SYN appears in the packet capture on the WAN interface.

This happens for all tested sites

• which are located at Cloudflare
• which are not excluded from HTTPS inspection
• which haven't been visited in the last hour

The CA list on the gatewway is up to date and complete.

Any thoughts what or which blade could cause the delay?

Could a WSTLSD daemon debug help?

Jas Man

[Timothy_Hall]

• Verify that all the DNS servers defined in the gateway's Gaia OS are correct and responding promptly using nslookup or dig.
• Try temporarily disabling Anti-bot on the gateway. If the problem goes away, it is likely due to the new DNS protections that I've seen cause delays like this.
• Make sure the "HTTP/HTTPS Proxy Server" feature is not enabled on the "HTTP/HTTPS Proxy" screen of the gateway
  object.
• This can also be caused by OSCP/CRL retrieval failures or delays, see here: https://community.checkpoint.com/t5/Maestro/Unreached-OCSP-causes-serious-lag/m-p/245427#M3265
• Are you using DNS over HTTPS, especially in Firefox?  sk183628: HTTPS Inspection is slow when using the Firefox web browser with DoH enabled

If still no joy, please post the output of enabled_blades run from the gateway along with your code and JHFA level.

[JasMan]

The defined DNS sever are responding immediatly and correct.
HTTP/HTTPS Proxy is disabled.
The delay is not caused by any TLS-related issue, because the delay happens during the first SYN. 

Enabled blades: fw vpn urlf av appi ips identityServer SSL_INSPECT anti_bot mon

I'll try to disable the AntiBot blade later. It's currently not possible.

[the_rock]

Is it same on every browser? If so, you can always try use secure DNS setting, see if it makes any difference. I believe in every browser, there are few options, ie google dns, cloud flare, etc.

Andy

[the_rock]

I did more checking into this and saw case I had with customer and what I attached turned out to tbe the issue, it was on hold and when we changed to background, all worked well. Might be worth checking.

Andy

[JasMan]

Yes it happens in all browsers. DNS seems to be fine, because the delay happens during the SYN at the perimeter firewall on the outgoing interface. 
"Unfortunately" categorization is already set to "Background".

[the_rock]

K, then maybe disabling AB blade is not a bad idea.

Andy

[Timothy_Hall]

My guess is still something with the two new DNS protections in Anti-bot.  Unfortunately, granular control of these features is not exposed in R81.20, but it is possible to toggle these two DNS protections in R82 individually without completely disabling Anti-bot.  Also see the discussion of "dns=bg" here: sk92224: Optimizing the categorization of DNS traffic by changing the Resource Classification Mode, ...

The fact that websites perform fine for 60 minutes after the initial delay in access, and then the initial delay returns after 60 minutes again, screams some kind of cache issue to me, possibly CRL/OCSP, but it seems too early in the connection for that.  Are you using FQDN Domain objects in your Access Control policy or HTTPS Inspection policy?  The cache timer for domain entries is 60 minutes, so I'm wondering if when that expires, there is a long delay having to retrieve the FQDN again and get it in the cache, but then once the site's domain gets into the cache, life for that site is good for 60 minutes, until it times out of the cache and the gateway has to retrieve it again, which causes the delay. 

Is DNS traffic subject to Geo Policy or Geo Updatable object blocking for any countries?  If Cloudflare's initial servers are located in a prohibited country it may take awhile before your DNS resolver hits upon a server that is in an allowed country and not blocked.

[JasMan]

I fully aggree with you. It must have to do with a caching function. And AntiBot sound to be a valid culprit. Will test to toggle the protection settings. But not on a Friday....you know why 🙂

The responsible policies are not using FQDN domain objects.

As far as I know there's no geo-based policy and/or object. But I've to check this.

DNS was my favorite suspect at first.

I've captured all DNS queries on the firewall during the issue happened to check the DNS response times. But there was no query for the affected domain.
I would guess that the gateway doesn't do DNS queries for HTTP/HTTPS connection. Because the client already resolved the domain to an IP address. The firewall get the connection request from the client to the public IP address. There's no need for the firewall to resolve any domain. And when it comes to HTTPS inspection the firewall can get the URL out of the SNI.
Does it makes sense?

[PhoneBoy]

It's not clear if wstlsd is relevant here since this problem is happening on the SYN (which has no data for wstlsd to work with).
I would get TAC involved.