Agent Readiness Compare

Setting a policy for AI crawlers and agents

Training, search and assistants are three decisions, not one.

In short

Decide separately whether to allow AI training crawlers, search crawlers and assistants that fetch pages for a user. Express each decision in robots.txt groups and enforce what matters at your CDN. Use verified bot identity rather than user agent names where you can, and decide whether any content should be paid for rather than free or blocked.

Three decisions

  • Training: may AI companies use your content to train models?
  • Search: may AI search and answer engines index your pages?
  • Assistants and agents: may an agent acting for a user, such as a prospective customer, fetch and use your pages?

Blocking everything answers all three with no, including the last, which turns away agents working for your buyers. AgentReady v2.0 makes not blocking agents a MUST (AR-FIND-01). Ora's lab found (19 Oct 2026) that a 403 for agent user agents stopped retrieval entirely in its test, a vendor claim from one docs site. Write the three decisions down separately first.

Expressing the policy

robots.txt lets you write a group per user agent, so a site can allow search crawlers and assistants and disallow a training crawler by name. RFC 9309 is clear that these rules "are not a form of access authorization". Well-behaved crawlers follow them and others may not.

Content Signals add a way to state preferences for training, search and AI input. Cloudflare reported on 17 April 2026 that 4 percent of the 200,000 most visited domains declared them. CDNs add settings on top. On 15 September 2026 Cloudflare introduced a "Disallow AI Training" setting that uses robots.txt Disallow rules to opt out of training while staying discoverable in search.

Cloudflare's Markdown for Agents sends a default content-signal: ai-train=yes, search=yes, ai-input=yes header unless the origin overrides it. Check that header against your policy before enabling the feature.

Enforcing it

Enforce at the edge, with verified identity. User agent strings can be copied, so a rule that trusts a name also trusts anything that claims it. Web Bot Auth lets a CDN check a signature instead. Cloudflare documents it as a bot verification method, and Vercel's bot verification supports it. Cloudflare's AI Crawl Control shows which AI services crawl a site and lets you allow or block each crawler.

Content you might charge for

A third option sits between allow and block: charge. Cloudflare's Pay per crawl, part of AI Crawl Control, was in private beta when this page's sources were read in September 2026. For agents that buy data or API calls per request, x402 uses HTTP 402 so a client can pay and retry. Both are commercial decisions. Record them in the same written policy.

Keeping the policy current

Crawler and agent names change. Confirm current names on each operator's own documentation, review the policy whenever you add a CDN rule, and re-test that the agents you want can still reach your key pages, as lesson 8 describes.

Sources

  1. RFC 9309, Robots Exclusion Protocol. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-09.
  2. Cloudflare, accountable mixed-use AI crawlers. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/, read 2026-09.
  3. Cloudflare, Markdown for Agents docs. https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/, read 2026-09.
  4. Cloudflare, Web Bot Auth docs. https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/, read 2026-09.
  5. Vercel, verified bots. https://vercel.com/docs/botid/verified-bots, read 2026-09.
  6. Cloudflare, AI Crawl Control docs. https://developers.cloudflare.com/ai-crawl-control/, read 2026-09.
  7. Cloudflare, Pay per crawl docs. https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/, read 2026-09.
  8. x402.org. https://www.x402.org/, read 2026-09.
  9. Cloudflare, Agent Readiness. https://blog.cloudflare.com/agent-readiness/, read 2026-11-04.
  10. Ora, AgentReady launch post. https://ora.ai/blog/agentready-launch, read 2026-11-04.

Last reviewed 2026-11-05