robots.txt
In short
robots.txt: The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309. Example, read on 4 November 2026: Radar found 78 percent of the top 200,000 domains had a robots.txt. Cloudflare's post puts robots.txt, sitemap.xml and Link response headers under the Discoverability dimension of its readiness score.
Definition
The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309.
A sourced example
Radar found 78 percent of the top 200,000 domains had a robots.txt. 1
Where the term appears on this site
In the profiles and sources
- Cloudflare's post puts robots.txt, sitemap.xml and Link response headers under the Discoverability dimension of its readiness score.1
Standards
- Content SignalsA Content-Signal line in robots.txt that states preferences for ai-train, ai-input and search use of a site's content. 1
- RFC 9309 Robots Exclusion ProtocolThe Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. 2
The other 39 terms are in the glossary index.
Sources
- Cloudflare: Introducing the Agent Readiness score (2026-04-17). https://blog.cloudflare.com/agent-readiness/, read 2026-11-05.
- RFC 9309. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-11-04.
Last reviewed 2026-11-05