Continuous accountability for Verified bots

Cloudflare announced a new system for continuously evaluating the behavior of Verified bots, moving its verification program beyond identity checks toward ongoing behavioral accountability. The company says it is already using the system internally to assess how consistently Verified bots honor website preferences, and that it plans to make it available to all Cloudflare customers in the future, at which point customers will be able to choose the trust threshold used to decide how Verified bot traffic is handled.
The framing Cloudflare uses is that verifying a bot's operator answers "are you who you say you are?" but not "can we trust you?" Being a Cloudflare Verified Bot broadly requires meeting three criteria: identifying yourself accurately and securely, doing only what you say you will do, and not engaging in abusive behavior. Bots use Web Bot Auth to sign requests that establish identity, and BotBase catalogues them in an open format. The new system targets the third criterion by measuring behavior over time rather than at a single point.
Cloudflare draws a deliberate distinction between risk and trust. Risk is the likelihood that a particular request or point-in-time action may be harmful and can change moment to moment. Trust, in this model, is built over repeated and consistent behavior. As an illustrative case, Cloudflare notes that it de-listed Perplexity's bot from its Verified bot list in 2025 after the bot sought to access resources on sites that had opted out of scraping and then attempted to circumvent controls when those requests were blocked.
The initial signals the system evaluates are whether a bot respects instructions and whether it responds to direct feedback. Cloudflare describes these as common-sense rules and a courtesy to the operators who provide the information bots seek. Rather than producing a permanent verdict on a bot or its operator, the observations translate into an explainable trust band: a range showing how consistently a bot respects site preferences along with the observed behaviors that informed the assessment.
For site-preference compliance, Cloudflare's system watches whether a bot's observed traffic is consistent with published robots.txt preferences over time, in line with RFC 9309. The company says isolated violations and one-off events may not be significant; what it looks for instead are repeated patterns that disregard site preferences. Cloudflare also stresses a narrow reading of robots.txt: the file sets minimum rules for what bots should not do, but silence in robots.txt does not imply that anything unstated is permitted, and a site can communicate that continued access is unwelcome through other signals.
For load and rate behavior, Cloudflare points to HTTP 429 responses as a clear way for a site to push back. A 429 tells a visitor to reduce its request rate and sometimes indicates how long to wait before retrying. The system checks whether a bot's traffic adapts after receiving that signal: does it slow down, pause, or keep sending requests at the same pace? Consistently slowing down or pausing raises the trust band over time, Cloudflare says, while repeatedly ignoring the signal lowers it.
The system is positioned as one piece of a broader accountability loop alongside two existing Cloudflare products. Bot Preference Sync helps site owners publish their preferences, the behavioral observations show whether Verified bots consistently honor those preferences over time, and Bot Management lets site owners decide how automated traffic is handled. Cloudflare's stated aim is that trust be earned through transparent, sustained behavior rather than assumed from identity alone.
The rollout is staged. Cloudflare says it is in an initial phase building the behavioral history needed to assess sustained patterns rather than isolated events, and refining how observations become useful guidance. Public scores on Radar are described as a future step, as is expanding the ways behavior is measured. The company also says responsible bot operators gain a way to demonstrate that trust is being upheld over time.
For developers and product teams running crawlers or agents, the practical implication is that Verified bot status is becoming a continuously maintained state rather than a one-time credential. A likely trade-off is added operational discipline: crawler teams may need to make rate-limit handling and robots.txt compliance first-class, monitored behaviors rather than best-effort conventions, since repeated lapses could affect how their traffic is treated. Because the customer-facing thresholds are still forthcoming, the precise consequences of a lower trust band remain unspecified.
Why it matters: site owners and bot operators both get a shared, behavior-based signal instead of relying on identity alone, and Cloudflare customers will eventually control the threshold at which Verified bot traffic is treated differently. Crawler and agent teams should treat RFC 9309 compliance and 429 handling as measurable requirements, not courtesies, since sustained patterns—not isolated incidents—are what feed the trust band. This is my inference from the announced design, as the operational details of customer thresholds have not yet been published.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Identity alone doesn't guarantee a bot will behave responsibly. We are introducing a system to continuously evaluate whether Verified bots honor site preferences, like robots.txt and HTTP 429s, ensuring they remain accountable over time.