Agent Readiness Compare

robots.txt

In short

robots.txt: The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309. Example, read on 4 November 2026: Radar found 78 percent of the top 200,000 domains had a robots.txt. Cloudflare's post puts robots.txt, sitemap.xml and Link response headers under the Discoverability dimension of its readiness score.

Definition

The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309.

A sourced example

Radar found 78 percent of the top 200,000 domains had a robots.txt. 1

Where the term appears on this site

In the profiles and sources

  • Cloudflare's post puts robots.txt, sitemap.xml and Link response headers under the Discoverability dimension of its readiness score.1

Standards

The other 39 terms are in the glossary index.

Sources

  1. Cloudflare: Introducing the Agent Readiness score (2026-04-17). https://blog.cloudflare.com/agent-readiness/, read 2026-11-05.
  2. RFC 9309. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-11-04.

Last reviewed 2026-11-05