Agent Readiness Compare

Six agent readiness myths, checked against the specifications

Each claim, what the source says, and what to do instead.

In short

llms.txt does not control crawling, robots.txt does not enforce anything, a user agent string proves nothing, WebMCP is not a version of MCP, most sites do not need x402, and a readiness score is not a measure of AI visibility or revenue. Each section below names the source that settles the point.

Myth 1: llms.txt controls what AI crawlers can fetch

It does not. llms.txt, proposed in 2024 and maintained through llmstxt.org, is a markdown file with a required H1, an optional summary and lists of links; its job is to point language models to the pages that matter. Crawl permissions live in robots.txt. Version 2 of the proposal, dated 10 August 2026, adds link relations but still does not grant or block access.

Ora's lab found (19 Oct 2026) that in a controlled experiment on one docs site, adding llms.txt did not change whether either test client retrieved the answer. That is a vendor finding on one site, not a general rule. What to do: publish both files and keep them consistent. See the llms.txt standard page.

Myth 2: robots.txt blocks AI agents

robots.txt asks; it does not enforce. RFC 9309 states that its rules "are not a form of access authorization." Well-behaved crawlers follow them, and others may not. What to do: use robots.txt to state your policy, and enforce what matters at your server or CDN, for example with Cloudflare AI Crawl Control or verified bot rules. See the RFC 9309 standard page.

Myth 3: a user agent string tells you which bot is visiting

Any client can send any User-Agent header, so a rule that allows GPTBot by name also allows anything that claims the name. Verification needs a signature. Web Bot Auth builds on HTTP Message Signatures (RFC 9421), with the operator's public keys in a key directory at /.well-known/http-message-signatures-directory. Cloudflare documents it as a bot verification method. Web Bot Auth is still an IETF draft, so treat it as a direction, not a settled standard. See the Web Bot Auth standard page.

Myth 4: WebMCP is the browser version of MCP

They share the idea of described tools, but they are separate efforts. MCP is a protocol between hosts, clients and servers using JSON-RPC 2.0, with a versioned specification whose latest revision is dated 2026-07-28. WebMCP is a proposal from the W3C Web Machine Learning Community Group that lets a page register tools, currently through document.modelContext, which an in-browser agent calls using the user's session. It is not a W3C Standard. What to do: choose by where the task lives. See the MCP standard page and the WebMCP standard page.

Myth 5: every agent-ready site needs x402

x402 fits resources an agent should buy per request, such as per-call APIs or datasets: the server answers an unpaid request with HTTP 402, and the client pays and retries. Most subscription software is bought by companies on contracts, invoices or cards, so readable pricing and a checkout or demo request an agent can complete matter more. In Ora's catalog all six Payments checks are bonus checks, so a site without payments is not penalised there. What to do: decide per resource and record the decision. See the x402 standard page.

Myth 6: a high readiness score means more AI visibility or more sales

A readiness score summarises which signals a scanner found. It is not designed to measure how often you appear in AI answers, and no scanner profiled here measures sign-ups or revenue.

Ora's lab found (8 Oct 2026), over 37,927 runs on 1,056 business sites, that agents clearly recommended agent-ready businesses in 20 percent of answers against 11 percent for others. Those are Ora's findings and this site has not tested them. What to do: treat readiness, AI visibility and business outcomes as three separate measurements, and attribute outcomes with your own analytics. See AX versus AEO and how to read an Ora score.

The myths persist because the files and scores look alike

The files and protocols arrived quickly and look alike: two text files at the site root, two ways to expose tools and several scores with similar names. The fix is to go back to the source each time. Every standard page on this site states its maintainer, its status and a link to the specification, and the FAQ collects the short answers.

Sources

  1. llmstxt.org. https://llmstxt.org/, read 2026-09.
  2. llms.txt changes. https://llmstxt.org/changes.html, read 2026-09.
  3. RFC 9309, Robots Exclusion Protocol. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-09.
  4. RFC 9421, HTTP Message Signatures. https://www.rfc-editor.org/rfc/rfc9421.html, read 2026-09.
  5. Cloudflare, Web Bot Auth documentation. https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/, read 2026-09.
  6. MCP specification. https://modelcontextprotocol.io/specification/latest, read 2026-09.
  7. WebMCP repository. https://github.com/webmachinelearning/webmcp, read 2026-09.
  8. x402.org. https://www.x402.org/, read 2026-09.
  9. Cloudflare, AI Crawl Control documentation. https://developers.cloudflare.com/ai-crawl-control/, read 2026-09.
  10. Ora, check catalog. https://ora.ai/api/checks, read 2026-11-04.
  11. Ora, AgentReady launch post. https://ora.ai/blog/agentready-launch, read 2026-11-04.
  12. Ora, AX is the new AEO. https://ora.ai/blog/ax-is-the-new-aeo, read 2026-11-04.

Last reviewed 2026-11-05