Agent Readiness Compare

How websites recognise AI agents: user agents, robots.txt and signed requests

Three signals, and which of them is verification.

In short

A site has three ways to tell which bot or agent is visiting: the user agent it claims, the rules it is asked to follow in robots.txt, and cryptographic signatures that prove who sent a request. Only signatures are verification. The other two rely on the client being honest.

The user agent is a claim

Every HTTP request can carry a User-Agent header, such as GPTBot or ClaudeBot for AI crawlers. It is a claim, not proof. Any client can send any string, so user-agent rules alone cannot separate a real AI crawler from automation that copies its name.

What robots.txt does and does not do

robots.txt, standardised as RFC 9309 in September 2022, lets a site publish path rules per user agent, and well-behaved crawlers follow them. The RFC states that "these rules are not a form of access authorization". robots.txt asks. It does not enforce.

On 15 September 2026 Cloudflare added a setting that uses robots.txt Disallow rules to opt out of AI training while staying in search. It relies on crawler operators it designates as "Accountable" to honour the rules.

How signed requests work

Web Bot Auth builds on HTTP Message Signatures (RFC 9421). The bot operator publishes public keys in a key directory at a well-known path and signs each request. The site, or its CDN, checks the signature against those keys. Cloudflare's documentation lists the Internet-Drafts it implements and uses Ed25519 keys. Vercel's bot verification also supports Web Bot Auth.

Ora's scanner is a public example. It identifies as OraBot/1.0 and signs its requests with an Ed25519 key under the Web Bot Auth profile, and Ora serves its key directory at /.well-known/http-message-signatures-directory. A few test requests that copy other crawlers' user agents are left unsigned on purpose.

What a site should do with a verified agent

Verification only answers who sent the request. The site still decides what that agent may do. Common choices are to let verified agents through a bot challenge, rate-limit them separately from unknown automation, and log them for analytics. Lesson 9 covers the policy side.

The stakes are practical. AgentReady v2.0 makes AR-FIND-01 a MUST: do not block agents in robots.txt or at the network layer. Ora's lab found (19 Oct 2026) that returning 403 to agent user agents scored 0 of 5 for both clients it tested, a vendor claim from a single docs site.

Agents acting for a signed-in user

A signature identifies the operator, not the person. When an agent acts inside a user's account, access is usually delegated with OAuth 2.0 scopes. The MCP specification of 28 July 2026 tightened its authorization rules, and the MCP roadmap of 22 August 2026 lists agent identity as a priority.

Sources

  1. RFC 9309, Robots Exclusion Protocol. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-09.
  2. RFC 9421, HTTP Message Signatures. https://www.rfc-editor.org/rfc/rfc9421.html, read 2026-09.
  3. Cloudflare, Web Bot Auth docs. https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/, read 2026-09.
  4. Vercel, verified bots. https://vercel.com/docs/botid/verified-bots, read 2026-09.
  5. Cloudflare, stay discoverable in search while disallowing AI training. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/, read 2026-09.
  6. MCP blog, the 2026-07-28 specification. https://blog.modelcontextprotocol.io/posts/2026-07-28/, read 2026-09.
  7. Ora, OraBot and bot policy. https://ora.ai/bot, read 2026-11-04.
  8. Ora, AgentReady launch post. https://ora.ai/blog/agentready-launch, read 2026-11-04.

Last reviewed 2026-11-05