When does Cloudflare crawl my site?
Cloudflare may crawl or make HTTP requests to your site to make sure its protected and performing properly.
Crawling situations
Specific products
Cloudflare will crawl your site when you have specific products enabled:
- Always Online
- User-Agent:
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)
- User-Agent:
- SSL/TLS recommender
- User-Agent:
Cloudflare-SSLDetector
- This crawler ignores your
robots.txt
file unless there are rules explicitly targeting the user agent.
- User-Agent:
- Load balancing monitors
- User-Agent:
Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: <POOLID>)
- User-Agent:
- Health checks
- User-Agent:
Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; healthcheck-id: <HEALTHCHECK_ID>)
- User-Agent:
- Security Insights
- User-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)
- User-Agent:
Other situations
Cloudflare will also crawl your site in other, specific situations:
- Speed tests
- User-Agent:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 PTST/190628.140653
- Triggered when: You launch a speed test from within the Cloudflare dashboard.
- User-Agent:
- Support diagnostics:
- User-Agent:
Cloudflare-diagnostics
- Triggered when: Cloudflare support engineers perform error checks and by continuous monitoring used to raise intelligent alerts in the Cloudflare dashboard.
- User-Agent:
- Certificate validation:
- User-Agent:
Cloudflare Custom Hostname Verification
- Triggered when: You choose to validate a custom hostname certificate with a DCV HTTP token.
- User-Agent: