Understand site traffic
3 min read
Another characteristic to consider is your application’s traffic. Several aspects of your traffic might affect how you implement Bot management.
Bot attacks
Situations
If you are experiencing bot attacks, consider the nature of the attack.
These types of actions fall into two general categories:
A bot submitting data through a form
- Credential stuffing on login endpoints
- Content spam
- Inventory hoarding
- Credit card stuffing
A bot viewing data intended for human eyes only
- Content scraping
- Ad click fraud
- Email address harvesting
Implementation details
If bots are submitting data through forms, you should likely be creating focused rules to block traffic on specific endpoints.
If bots are viewing data intended for human eyes only, you may want broader rules limiting bot interactions across your application.
Verified bots
Situations
Cloudflare maintains an internal list of Verified Bots that are associated with search engine optimization (SEO), website monitoring, and more.
You can use this list to prevent any bot protection measures from impacting otherwise helpful bots, such as search crawlers.
For a partial list of verified bots, refer to Cloudflare Radar.
Implementation details
Verified bots are blocked by default when you create firewall rules using cf.bot_management.score
.
To allow verified bots, you would need to include not (cf.bot_management.verified_bot)
as part of a firewall rule.
Automated traffic
Situations
By default, Bot management rules will block non-browser traffic.
This can be problematic if your application receives a lot of API traffic, or if you have tools related to:
- Indexing content for search.
- Auditing content (links, headers, etc.).
- Monitoring uptime.
- Forward proxying traffic, such as secure web gateways.
Implementation details
Depending on your application, you may want to write rules that allow specific types of automated traffic or rules that allow all automated traffic to specific endpoints (/api
, for example).
In some cases, APIs might be better suited for API Shield than Bot Management.
You should also take time to review Bot analytics to make sure you fully understand the automated traffic reaching your site. Often, you might discover services maintained by a different team or other surprises.
Pay specific attention to:
- Which endpoints are being targeted.
- The top non-Mozilla user agents.
- Traffic from Outlook or Office user-agents.
- Traffic from cloud-based Secure Web Gateways (ASNs labeled with the proxy provider).
- Traffic from on-premises forward proxies.
- Whether requests come from a predictable IP address and ASN, or have a similar JA3 fingerprint.
Mobile app traffic
Because of how mobile applications send requests, Bot Management has the potential to score mobile traffic differently than browser-based traffic.
Implementation details
Review analytics
Take extra time to review Bot analytics to evaluate how your mobile application traffic is performing.
You can generally identify mobile traffic with common user agent strings, though these strings may differ between iOS and Android. Malicious actors might also try to impersonate your mobile application traffic with user agent strings.
Cloudflare recommends being as specific as possible when analyzing traffic and creating rules, usually including a combination of user-agent values, IP addresses or ASNs, and JA3 fingerprints.
Adjust rules
If your application uses mTLS, you can also integrate that certificate’s presence into your firewall rules using the cf.tls_client_auth.cert_verified
field.
Native or progressive web applications should also only use Block as a firewall rule action.
Unit 5 of 5