How AI agents read a web page: HTML, markdown and llms.txt
What arrives in the response, and three checks to run first.
In short
An agent reads what your server returns. If key facts appear only after JavaScript runs, a plain fetch never sees them. Server-rendered HTML is the baseline and the first AgentReady MUST for reading. A markdown version of each page and an llms.txt file make reading cheaper, but they do not replace readable HTML.
What an agent receives
When an agent fetches a URL it gets the HTTP response: headers and a body. Some agents render JavaScript in a browser. A plain fetch does not. A pricing table injected by client-side code may be empty in the response the agent reads. Server-rendering or prerendering pricing, plans, docs, contact and sign-up pages removes that risk.
AgentReady v2.0 makes this a MUST: AR-READ-01 asks for the answer in the initial HTML, and AR-READ-02 asks for accurate status codes. Ora's lab found (19 Oct 2026) that hiding the answer in JavaScript scored 0 of 5 for plain fetch, one of only two configurations that stopped retrieval in its 190-probe test. That is a vendor claim.
Markdown negotiation
A client can ask for markdown by sending Accept: text/markdown. A server or CDN that supports it returns a markdown version of the page instead of HTML. Cloudflare's Markdown for Agents does this conversion at the edge. Its launch post of 12 February 2026 describes the request header and an x-markdown-tokens response header with an estimated token count. Markdown drops layout markup, so the agent spends its context on content.
Cloudflare reported up to 80 percent fewer tokens for markdown than HTML in some cases, which is a vendor claim. Ora's lab found (19 Oct 2026) that agents asked for markdown on about 65 percent of fetches, but markdown mirrors and negotiation never changed whether retrieval succeeded. Markdown lowers the cost of a read. It does not rescue a page the agent could not fetch.
What llms.txt adds
llms.txt is a markdown file at the site root with a title, a short summary and lists of links to the pages that matter most. It is a curated index, not a sitemap. The proposal was revised to v2 in August 2026. Pages can now point to their markdown version (rel="alternate" type="text/markdown") and to the llms.txt that covers them (rel="describedby"). Both page.html.md and page.md URL forms are accepted, and an llms.txt in a subpath covers the pages under that path.
Ora's lab found (19 Oct 2026) that 86 percent of llms.txt fetches arrived through a link, not a guessed path. Linking the file from your pages matters more than placing it at the root. AgentReady keeps llms.txt at SHOULD. The llms.txt standard page has the detail.
Structured data
JSON-LD states facts in a form machines parse without guessing: the organization, its products, and offers with prices. It sits in the page's HTML, so it survives when an agent cannot run JavaScript. Cloudflare's Markdown for Agents keeps JSON-LD as a fenced code block when it converts a page. JSON-LD does not replace readable text. Prices and plan limits should also appear in the page body, not only in images, sliders or toggles.
Three checks to run first
- Fetch your pricing page with JavaScript disabled and confirm the plans and prices are in the HTML.
- Request a key page with
Accept: text/markdownand look at what comes back. - Open /llms.txt and confirm every link resolves to a current page.
Sources
- llmstxt.org. https://llmstxt.org/, read 2026-09.
- llms.txt changes. https://llmstxt.org/changes.html, read 2026-09.
- Cloudflare, Markdown for Agents launch post. https://blog.cloudflare.com/markdown-for-agents/, read 2026-09.
- Cloudflare, Markdown for Agents docs. https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/, read 2026-09.
- Cloudflare, Agent Readiness. https://blog.cloudflare.com/agent-readiness/, read 2026-11-04.
- Ora, AgentReady launch post. https://ora.ai/blog/agentready-launch, read 2026-11-04.
- AgentReady specification, v2.0. https://github.com/agentready-org/standard/blob/main/spec/spec.md, read 2026-11-04.
Last reviewed 2026-11-05