Agent Readiness Compare

RFC 9309 Robots Exclusion Protocol

Layer: find and trust. Version: RFC 9309, September 2022.

In short

RFC 9309: The Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. Layer: find and trust. Maintained by IETF. Version: RFC 9309, September 2022. AgentReady AR-FIND-01, AR-FIND-03 map to it; Ora, Cloudflare scanner, agent-ready.dev check it.

What it is

The Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. 1

Specification: rfc-editor.org. Version or date: RFC 9309, September 2022.

Who maintains it

IETF.

Where it sits on the Standards Map

Layer: find and trust. See it among the other fifteen on the Standards Map.

What it does not do

Is voluntary for crawlers and says nothing about bot protection at the network layer, which AgentReady AR-FIND-01 also covers.

Which scanners check it

The evidence read on 2026-11-04 lists 3 of the 6 public scanners as checking it.

AgentReady identifiers

2 of the 24 AgentReady identifiers map to this standard, including 1 at MUST level: AR-FIND-01. The mapping is this site's reading of the spec and each scanner's check list.

  • AR-FIND-01 (MUST)
  • AR-FIND-03

Glossary terms

Sources

  1. RFC 9309. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-11-04.

Last reviewed 2026-11-05