---
title: "robots.txt: definition and example | Agent Readiness Compare"
description: "robots.txt: The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309. Sourced example, read 4 November 2026."
canonical: https://agentreadinesscompare.com/glossary/robots-txt/
site: "Agent Readiness Compare"
last_updated: 2026-11-05
section: "Glossary"
---

# robots.txt: definition and example | Agent Readiness Compare

> robots.txt: The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309. Sourced example, read 4 November 2026.

Source: https://agentreadinesscompare.com/glossary/robots-txt/
Site: Agent Readiness Compare
Last reviewed: 2026-11-05

# robots.txt

In short

robots.txt: The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309. Example, read on 4 November 2026: Radar found 78 percent of the top 200,000 domains had a robots.txt. Cloudflare's post puts robots.txt, sitemap.xml and Link response headers under the Discoverability dimension of its readiness score.

## 1. Definition

The file at a site root that tells crawlers which paths they may fetch, standardised in RFC 9309.

## 2. A sourced example

Radar found 78 percent of the top 200,000 domains had a robots.txt. [1](#src-1)

## 3. Where the term appears on this site

### In the profiles and sources

-   Cloudflare's post puts robots.txt, sitemap.xml and Link response headers under the Discoverability dimension of its readiness score.[1](#src-1)

### Standards

-   [Content Signals](https://agentreadinesscompare.com/standards/content-signals/)A Content-Signal line in robots.txt that states preferences for ai-train, ai-input and search use of a site's content. [1](#src-1)
-   [RFC 9309 Robots Exclusion Protocol](https://agentreadinesscompare.com/standards/rfc-9309/)The Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. [2](#src-2)

The other 39 terms are in the [glossary index](https://agentreadinesscompare.com/university/glossary/).

## 4. Related

-   [Glossary index](https://agentreadinesscompare.com/university/glossary/)

## 5. Sources

1.  Cloudflare: Introducing the Agent Readiness score (2026-04-17). [https://blog.cloudflare.com/agent-readiness/](https://blog.cloudflare.com/agent-readiness/), read 2026-11-05.
2.  RFC 9309. [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-11-04.

Last reviewed 2026-11-05
