Tuning best practices for crawlers

Eve_Z — Tue, 22 Jul 2025 19:27:41 GMT

Hello Community,

I am aware that CloudGuard WAF licenses are counted per number of HTTP/HTTPS requests. In some of our websites, most of the prevented traffic comes from GoogleBots IP addresses. These bots are crawlers that index the websites for search engines, but CloudGuard WAF classifies the traffic as malicious, prevents it and counts it from the license.

What tuning best practices do you recommend to reduce the amount of counted requests from our license?

Regards.

Re: Tuning best practices for crawlers

Danny — Tue, 05 Aug 2025 14:31:36 GMT

WAF security-practices

try these crawler tuning tips:

whitelist known crawlers
- create custom rules to allow traffic from verified crawler addresses (example list)
tune bot classification features in anti-bot protection to improve distinguish between good and malicious bots
adjust security practices
- activate web application protection and web API protection in detect/learn mode before switching to prevent mode, this allows the system to learn traffic patterns and reduce false positives
reduce inspection scope
- exclude static resources (images, css, .js) from inspection if they are frequently accessed by crawlers by configuring URL patterns or content types to bypass WAF inspection

topic Tuning best practices for crawlers in WAF

Tuning best practices for crawlers

Re: Tuning best practices for crawlers