RFC 9309 Robots Exclusion Protocol
Layer: find and trust. Version: RFC 9309, September 2022.
In short
RFC 9309: The Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. Layer: find and trust. Maintained by IETF. Version: RFC 9309, September 2022. AgentReady AR-FIND-01, AR-FIND-03 map to it; Ora, Cloudflare scanner, agent-ready.dev check it.
What it is
The Standards Track definition of robots.txt: user-agent groups, allow and disallow rules, error handling and caching. 1
Specification: rfc-editor.org. Version or date: RFC 9309, September 2022.
Who maintains it
IETF.
Where it sits on the Standards Map
Layer: find and trust. See it among the other fifteen on the Standards Map.
What it does not do
Is voluntary for crawlers and says nothing about bot protection at the network layer, which AgentReady AR-FIND-01 also covers.
Which scanners check it
The evidence read on 2026-11-04 lists 3 of the 6 public scanners as checking it.
AgentReady identifiers
2 of the 24 AgentReady identifiers map to this standard, including 1 at MUST level: AR-FIND-01. The mapping is this site's reading of the spec and each scanner's check list.
- AR-FIND-01 (MUST)
- AR-FIND-03
Glossary terms
Sources
- RFC 9309. https://www.rfc-editor.org/rfc/rfc9309.html, read 2026-11-04.
Last reviewed 2026-11-05