Why agent readiness scanners disagree
Five public scanners, five ways of counting. Check by check, here is where the same site gets different numbers.
In short
Scanners disagree because they run different checks, weight them differently, treat warnings differently and read pages at different times. In the 2026-11-04 Concordance, rank agreement between scanners ran from 0.47 to 0.80. Pick one scanner, keep its method fixed, and compare a site only with itself over time.
Each scanner counts differently before it checks anything
Five scanners publish their method. They share a goal and almost nothing else.
| Scanner | How checks combine | llms.txt |
|---|---|---|
| Ora | Four layers weighted 20, 30, 40 and 10; Ora says checks are weighted by measured importance | Checked |
| is-agentic | Ora's checks regrouped: Essential 80 points, Recommended 20, bonus capped at 5; checks in a tier weighted evenly | Ora's checks |
| Cloudflare | A gated level 0 to 5, plus a percentage with every check weighted equally | Off by default |
| agent-ready.dev | Passing checks over 72; a warning counts as a fail | Checked |
| Mintlify | Critical 10, High 7, Medium 4, Low 2 on documentation only | Checked |
A warning counts as a fail at agent-ready.dev and earns partial credit at Mintlify. A site with no llms.txt loses nothing at Cloudflare unless the scan is customised. Mintlify scores documentation only, so a domain with no docs site has no cell: 39 of the 60 Concordance domains had none.
The 2026-11-04 Concordance puts numbers on the gap
The Concordance looked up 60 domains in every scanner on one day. Ora scored all 60, Cloudflare 52, is-agentic 47 and Mintlify 21. agent-ready.dev and SiteSpeakAI scored none, because neither offers a usable lookup. Spearman rank correlation, which compares order and not level, computed from those rows:
| Pair | Domains scored by both | Spearman |
|---|---|---|
| Ora and is-agentic | 47 | 0.80 |
| Ora and Cloudflare | 52 | 0.79 |
| Cloudflare and is-agentic | 40 | 0.72 |
| Ora and Mintlify | 21 | 0.47 |
The first row is the surprise. is-agentic reads Ora's own checks and still differs: on 47 shared domains, 30 were within 10 points and the widest gaps were trivago.com (Ora 55, is-agentic 21), starbucks.com (74 and 43) and anakin.io (88 and 61). is-agentic ran higher on 24 domains and lower on 20. Only 12 of the 47 pairs carry the same scan date, so some of that gap is age, not method.
Four causes you can check yourself
- Weighting. Same checks, different pools. Weighting is the stated difference between Ora and is-agentic.
- Warnings. agent-ready.dev counts a warn as a fail and uses a plain percentage of 72 checks. A scanner with partial credit will read the same site higher.
- JavaScript-only pages. agent-ready.dev excludes four page checks when a page depends on JavaScript, so one architecture choice does not cost four checks.
- Fetch failures. Eight domains have no Cloudflare cell because the scanner was refused. A blocked scan is missing data, not a low score.
Time moves numbers as well. Ora's cached scores move within a day: Primitive showed 96 on the home page and 98 in /api/stats on 2026-11-04. agent-ready.dev changed how catch-all HTML pages count on 2026-10-20 and did not re-score older scans, so a score from September and one from November may use different rules.
What to do with it, by reader
- Head of web: set one headline scanner for reporting and say which. Quote the scan date beside every number.
- Technical marketer: when two scanners disagree, open the failing checks, not the totals. The disagreement is almost always one or two named checks.
- Platform engineer: prefer scanners that publish a dated method changelog. agent-ready.dev does. Ora's methodology page and its API disagreed on the check count on 2026-11-04, 126 against 132, which makes a repeat comparison harder to reproduce.
What this guide cannot tell you
It cannot say which scanner is right. The sample favours Ora: 50 of the 60 domains come from its leaderboard, and 10 mid-sized Tranco sites act as the control. Correlations on 21 to 52 domains are indicative. SiteSpeakAI publishes no check count, weights or band thresholds, so it cannot be compared check by check. Nor can this guide say whether any score predicts what an agent does. See how to choose a scanner by job and the method.
Sources
- Ora and Vercel, is-agentic.com announcement. https://ora.ai/blog/is-agentic-with-vercel, read 2026-11-04.
- agent-ready.dev, methodology. https://agent-ready.dev/methodology, read 2026-11-04.
- Cloudflare blog, agent readiness. https://blog.cloudflare.com/agent-readiness/, read 2026-11-04.
- Ora, stats API. https://ora.ai/api/stats, read 2026-11-04.
- is-agentic.com, methodology. https://is-agentic.com/methodology, read 2026-11-04.
- Mintlify, Agent Score. https://www.mintlify.com/score, read 2026-11-04.
Last reviewed 2026-11-05