How agents experience your site: the agent experience manifesto
Being surfaced in an AI answer is the start of the visit, not the end of it.
In short
Agent experience is how an AI agent gets on with your business once it has found you: whether it can read your pages, get past your bot rules and finish a task such as a sign-up, a demo request or a checkout. It covers what an agent needs at each step, where visits stop, how to test and ten principles.
The short version
Some visits to a website are now made by software acting for a person. Someone asks an assistant to compare three project tools, find a plan for a team of twelve, book a demo or buy a part. The assistant picks a few sites and then has to do the work itself: open the pricing page, read the plans, find the limits in the docs, fill in a form and sometimes pay.
Most talk about AI and websites covers the first half: whether your brand appears in the answer. That work is AEO. This page covers the second half. Once an agent has found you, can it read what you sell, get in and finish the job it was sent to do? We call that agent experience.
It is not one file or one protocol. It is the sum of choices you probably made for other reasons: how pages render, what bot rules allow, how forms are built, whether actions are reachable by a machine, and whether you can tell a legitimate agent from a scraper. Each can stop an agent that a human visitor would never notice.
One visit, layer by layer
Follow one visit in order. The five layers below are the ones in lesson 1. At each step there is something the agent needs and something that usually stops it.
Find. The agent first learns what exists and what it may fetch. robots.txt, standardised as RFC 9309, tells crawlers and agents which paths they may request. A sitemap points to pricing, product and docs pages. It needs a clear answer to two questions: where are the important pages, and may I read them? It is usually stopped by a blanket Disallow under User-agent: * written years ago for another purpose, or by a CDN rule that blocks what robots.txt allows. The agent cannot tell "not allowed" from "not there". AgentReady makes not blocking agents a MUST (AR-FIND-01).
Read. With a URL, the agent needs the content in a form it can parse. A plain fetch does not run JavaScript, so a pricing page that renders plans in the browser looks empty. AgentReady makes the answer in the initial HTML a MUST (AR-READ-01) and accurate status codes another (AR-READ-02). Markdown versions of key pages, through .md URLs or Accept: text/markdown, and llms.txt lower the cost of reading. Structured data in JSON-LD states facts an agent can quote. The usual stop is missing text, not missing markup: prices in an image, limits behind a toggle.
Act. Reading your pricing is not the goal. The person asked for a plan, a demo or a purchase. The agent either operates your interface like a person or calls a tool you expose. Operating the interface depends on labelled forms and steps that need no custom gesture. Calling a tool depends on a machine-readable description. MCP is how most agents call external tools and data, and AgentReady makes an MCP server or a CLI a MUST when agents should act on the product (AR-ACT-10). WebMCP, a Community Group draft, lets a page expose in-browser tools, and v2.0 raised it to SHOULD. agents.json describes flows that span several API calls. The usual stop is a form the agent cannot interpret, or an action that exists only as a button.
Pay. Some tasks end with a payment. x402 revives HTTP status 402 so a resource can answer an unpaid request with its payment requirements and an agent can pay and retry. UCP and ACP cover commerce checkout. Cloudflare's scanner reports x402, UCP and ACP checks but does not count them toward its score. AgentReady lists agent payments as MAY (AR-ACT-08). The usual stop is a checkout that assumes a person: a challenge the agent cannot pass, or a confirmation sent to a device it does not hold.
Trust. Identity runs through all the others. Anyone can claim to be a known crawler in a user agent string. Web Bot Auth uses HTTP message signatures (RFC 9421): the agent signs its requests, publishes its keys in a directory and your edge verifies the signature. Delegated access to a person's account should use OAuth, not shared passwords. The usual stop is bot protection that treats every automated request as hostile, so the agent sent by your own prospective customer is turned away with the scrapers.
Agents also act for other agents. A2A lets an agent publish an Agent Card so others can discover what it does. Discovery now reaches capabilities, not only pages. None of these layers is exotic. A careful web team already does most of them for accessibility, performance or security. The difference is that an agent meets all five in one visit, and a failure at any of them ends it.
Why people confuse agent experience with AEO
Partly because both start with "AI", and partly because both touch discovery. An AEO platform cares whether an answer engine can crawl and cite your pages. An AX tool cares whether those pages are reachable. On a slide the two can look like one category.
The vocabulary has also converged. Scrunch calls its product the Agent Experience Platform, and its Agent Pages serve agents a structured version of each page. Scrunch says the aim is that a brand appears more often in AI answers. That is a vendor claim about a reasonable product. Read the goal in it: appearing in answers. The measure is visibility, not whether the agent that arrives completes a sign-up.
The confusion changes what gets tested. If AX is treated as a subset of AEO, the team checks whether the brand is cited and stops there. The agent's visit after the citation, where the revenue sits, goes unexamined.
What AEO does well
Visibility in answers matters for discovery. If an assistant never mentions you, no agent is sent to your site and nothing downstream gets a chance. AEO platforms show which prompts mention a brand, which pages get cited and how that changes. Some also report which AI bots crawl a site, which is useful context for a crawler policy.
The limit is the question the tools are built to answer. They measure the answer engine. They are not designed to measure what happens on your site after the answer. Keep both reports and keep them separate. When they merge into one score, the part that is harder to fix tends to drop out of view. The reference page sets out the definitions and who sells what.
Where visits stop
The four cases below are composites written to show where a visit stops. They carry no measured figures.
- The cited SaaS with a JavaScript pricing page. A project tool is named in an answer and cited at its pricing page. The agent fetches the page and gets a header, a footer and a script tag. It reports that it found no pricing and recommends a competitor whose plans were in the HTML.
- The brand whose bot protection challenges every agent. An agent arrives to check stock and price. The edge returns a challenge it cannot pass, and the retailer does not verify signed bots. The agent moves to the next result.
- The store whose checkout needs a human step. The agent reads the price, adds the item and reaches a final step that needs a code sent to a phone it does not hold. The person is told to finish the purchase later.
- The demo form with unlabelled fields. Five inputs, no labels, placeholder text that disappears on focus, and an error shown only as an icon. The agent guesses and the form rejects it.
In each case the AEO work succeeded. The failure came afterwards, on the site, in places an AEO dashboard does not look. Ora's lab found (19 Oct 2026) a real-world echo: of 19 configurations of one docs site, only JavaScript-only answers and 403 responses to agent user agents stopped retrieval. That is a vendor claim from one site.
Surfaced and converted
We use two words for the two outcomes. Surfaced means an answer engine mentions and cites you. Converted, here, means the agent finished the task it was sent to do: the plan was compared accurately, the demo request was submitted, the account was created. These are working terms, not standards.
Keeping them apart keeps reporting honest. A rise in citations is a discovery result. A task run that now completes where it used to stop is an agent experience result. Neither proves the other, and neither is revenue without data from your own funnel. A team that tracks only being surfaced spends its effort on content that earns citations. A team that also tracks being converted finds a different list: render pricing on the server, label the form, align bot rules, describe the actions. Those jobs are usually smaller than a content programme, and each can be checked off.
Why task results can be re-checked
Answer engines generate an answer for each request. The same prompt asked twice can return different wording and different citations, and the models and sources change. A visibility figure is a sample. AEO platforms sample at volume on a schedule. Peec AI's Starter plan, for example, tracks 50 prompts for 95 USD a month (read 4 Nov 2026).
AX checks are tied to your own pages. A robots.txt rule allows a given agent or it does not. A pricing page returns plans in its HTML or it does not. A task run reaches the confirmation page or stops at a named step. The same file check on the same page gives a comparable result until the page changes. Task runs vary more between runs, so keep the task list short. A fix can be verified by re-running the check, and a regression can be traced to a release.
What to test instead
Start with published checks. A scanner that publishes its checks and grade bands lets you see why a score is what it is and which item to fix first. A score summarises signals found. It is not proof that a task works, so read the failing checks, not the number. The checklist summary outlines what the checks cover.
Then test the task, not only the file. Write down the three to five tasks an agent should complete: explain pricing, recommend a plan for a stated requirement, find a feature in the docs, submit a demo request, complete a checkout. Run an agent through each and keep the recording. Your first agent task test walks through a first run.
Then re-test after releases that touch pricing, sign-in, bot rules or checkout. Keeping readiness from regressing covers the routine. Finally, write a policy for which crawlers and agents you want, mirror it at the edge and review it when the list changes. Setting an AI crawler policy sets out the decisions.
Who does what
Tools in this space do different jobs, so this site profiles them by job rather than ranking them in one list. Only scanners are scored, under the method.
- Scanners. Ora scores a domain across four layers with a 132-check catalog and runs Journey tasks with a real agent. Cloudflare's scanner is free and reports a level from 0 to 5 on 23 checks. is-agentic.com is Ora's scan regrouped, so a result there is an Ora result. agent-ready.dev, SiteSpeakAI and Mintlify Agent Score check other sets.
- Agent-page products. Scrunch serves agents a structured page. Cloudflare Markdown for Agents converts HTML to markdown at the edge. Ora's fixing product is on a waitlist.
- AEO platforms. Profound, Peec AI and Otterly.AI measure how a brand appears in AI answers. They do not score whether an agent can read or use a site.
Ten principles of agent experience
Each principle can be tested again after the next release changes your site, your edge rules or the agents that visit.
- Write for the agent that has already found you. Discovery gets it to the door. The rest is your site's job.
- Test the task, not only the file. A file check shows which signals exist. A task run shows whether the agent finished.
- Put the facts in the HTML. Prices, plan limits and requirements belong in server-rendered text, not only in scripts, images or toggles.
- Make your policy match your edge. robots.txt and your CDN or firewall rules should say the same thing.
- Label everything a person would read. Forms an agent can fill in are forms with labels and accessible names.
- Give actions a machine-readable door. Expose the tasks that matter through an MCP server or a CLI, and through WebMCP where a page needs it.
- Verify, do not guess. Recognise signed agents with Web Bot Auth instead of trusting user agent strings or blocking all automation.
- Remove human-only steps from the paths that matter. If a verified agent cannot pass a step, that step ends the task.
- Re-test after every release that touches the journey. Pricing, sign-in, bot rules and checkout change, and so do agents.
- Keep surfaced and converted apart. Report citations as discovery and finished tasks as agent experience. Leave revenue to your own analytics.
Sources
- RFC 9309, Robots Exclusion Protocol. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-09.
- RFC 9421, HTTP Message Signatures. https://www.rfc-editor.org/rfc/rfc9421.html, read 2026-09.
- llmstxt.org. https://llmstxt.org/, read 2026-09.
- Model Context Protocol specification. https://modelcontextprotocol.io/specification/latest, read 2026-09.
- WebMCP specification. https://webmachinelearning.github.io/webmcp/, read 2026-09.
- agents.json specification. https://github.com/wild-card-ai/agents-json, read 2026-09.
- x402.org. https://www.x402.org/, read 2026-09.
- Cloudflare, Agent Readiness. https://blog.cloudflare.com/agent-readiness/, read 2026-11-04.
- Ora, AX is the new AEO. https://ora.ai/blog/ax-is-the-new-aeo, read 2026-11-04.
- Ora, AgentReady launch post. https://ora.ai/blog/agentready-launch, read 2026-11-04.
- AgentReady specification, v2.0. https://github.com/agentready-org/standard/blob/main/spec/spec.md, read 2026-11-04.
- Scrunch, agent experience platform. https://scrunch.com/platform/agent-experience, read 2026-11-04.
- Peec AI, pricing. https://peec.ai/pricing, read 2026-11-04.
Last reviewed 2026-11-05