---
title: "RFC 9309: standard reference | Agent Readiness Compare"
description: "RFC 9309 as a reference: maintainer, version, what it does not do, the AgentReady identifiers that map to it and which scanners check it."
canonical: https://agentreadinesscompare.com/standards/rfc-9309/
site: "Agent Readiness Compare"
last_updated: 2026-11-05
section: "Standards"
---

# RFC 9309: standard reference | Agent Readiness Compare

> RFC 9309 as a reference: maintainer, version, what it does not do, the AgentReady identifiers that map to it and which scanners check it.

Source: https://agentreadinesscompare.com/standards/rfc-9309/
Site: Agent Readiness Compare
Last reviewed: 2026-11-05

# RFC 9309 Robots Exclusion Protocol

Layer: find and trust. Version: RFC 9309, September 2022.

In short

RFC 9309: The Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. Layer: find and trust. Maintained by IETF. Version: RFC 9309, September 2022. AgentReady AR-FIND-01, AR-FIND-03 map to it; Ora, Cloudflare scanner, agent-ready.dev check it.

## 1. What it is

The Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. [1](#src-1)

Specification: [rfc-editor.org](https://www.rfc-editor.org/rfc/rfc9309.html). Version or date: RFC 9309, September 2022.

## 2. Who maintains it

IETF.

## 3. Where it sits on the Standards Map

Layer: find and trust. See it among the other fifteen on the [Standards Map](https://agentreadinesscompare.com/standards/#map).

## 4. What it does not do

Is voluntary for crawlers and says nothing about bot protection at the network layer, which AgentReady AR-FIND-01 also covers.

## 5. Which scanners check it

The evidence read on 2026-11-04 lists 3 of the 6 public scanners as checking it.

-   [Ora](https://agentreadinesscompare.com/tools/ora-ai/)
-   [Cloudflare scanner](https://agentreadinesscompare.com/tools/cloudflare/)
-   [agent-ready.dev](https://agentreadinesscompare.com/tools/agent-ready-dev/)

## 6. AgentReady identifiers

2 of the 24 AgentReady identifiers map to this standard, including 1 at MUST level: AR-FIND-01. The mapping is this site's reading of the spec and each scanner's check list.

-   AR-FIND-01 (MUST)
-   AR-FIND-03

## 7. Glossary terms

-   [robots.txt](https://agentreadinesscompare.com/glossary/robots-txt/)

## 8. Related

-   [Standards Map](https://agentreadinesscompare.com/standards/#map)
-   [Method](https://agentreadinesscompare.com/method/)

## 9. Sources

1.  RFC 9309. [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-11-04.

Last reviewed 2026-11-05
