How to run a proof of concept for an agent readiness scanner
A two-week plan: agree the criteria, run the tools on the same pages, and compare findings rather than grades.
In short
Agree what success means before you run anything, pick three pages and one task, and run each shortlisted scanner on the same targets in the same week. Compare the findings, the fixes they point to and how well each tool fits your workflow, not the headline grades. Record dates and methods so the write-up survives the next rule change.
This plan comes from an earlier experiment
Scanner grades are not comparable across tools: each has its own checks, weights and bands. The guide on choosing a scanner lists the questions to ask on paper. A proof of concept answers them on your own site, and shows your team what the output looks like before anyone reports a number to leadership.
Agree four things before day one
- The decision: what the scanner is for, such as finding gaps, gating releases in CI or reporting to leadership.
- The targets: three public pages and one task (next section).
- The criteria and their weights. The method page lists the seven this site uses: coverage, transparency, reproducibility, access, remediation, independence of method and freshness. Adjust the weights to your job.
- The owner and the end date.
Pick pages and a task an agent needs to act for a buyer
Choose the pricing page, one docs or feature page that answers a common question, and the page with your next step, such as a demo request or trial sign-up. For the task, pick one that matters commercially and needs no account, for example: explain our pricing and recommend a plan for a ten-person team. Write the expected answer down first.
Week one runs the tools and checks the findings
- Day 1: run each scanner on the three pages and save the full reports with the date.
- Day 2: run the task. If a tool records agent runs, as Ora's Journey does, use it. Otherwise run the task by hand with a browsing assistant and log each step.
- Day 3: put every finding in one table: page, finding, which tools reported it and which layer it belongs to.
- Days 4 and 5: check each finding by hand with the checklist, marking it confirmed or not reproduced.
Week two fixes findings and re-runs the tools
- Fix two or three confirmed findings, starting with the weakest layer.
- Re-run each scanner and the task, and note which tools picked up the fixes.
- If CI matters, try each tool's code path. Ora documents
npx ax audit, a REST API and an MCP server. is-agentic.com and agent-ready.dev document CLI or API routes too, and agent-ready.dev's API sits in its paid plan. - If you run on Cloudflare, check whether its edge features (Markdown for Agents, AI Crawl Control, Web Bot Auth verification) close findings without code changes.
Score each tool on what you saw
Score each tool against the criteria you agreed, using what you saw rather than what the vendor says. Useful measures: how many confirmed findings each tool reported, how many it missed, how many did not reproduce, whether it showed where the task failed and how long a re-scan took. Compare the headline grades last, as a sanity check. A tool that reports fewer, better findings can be more useful than one with a longer list.
Five traps spoil a proof of concept
- Comparing grades across tools. They use different scales.
- Testing only the home page. Agents need pricing, docs and forms.
- Forgetting login-gated flows. Scanners of this kind read public URLs, and Ora states that it does not scan private or logged-in pages, so test those flows separately.
- Changing the site mid-test without noting it. Date every change.
- Treating a score as an outcome. No scanner measures sign-ups or revenue.
The write-up fits on one page
Include the decision, the targets, the criteria and weights, a table of findings by tool, the task results, what was fixed and whether each tool detected the fix, and a recommendation. Attach the dated reports. If the choice is close, say so. Many teams end up with one tester and one edge provider, because the two do different jobs.
Sources
- Ora, methodology. https://ora.ai/methodology, read 2026-09.
- Ora, llms.txt. https://ora.ai/llms.txt, read 2026-09.
- Ora, llms-full.txt. https://ora.ai/llms-full.txt, read 2026-11-04.
- is-agentic.com. https://is-agentic.com/, read 2026-11-04.
- agent-ready.dev, comparison with alternatives. https://agent-ready.dev/agent-ready-vs-alternatives, read 2026-11-04.
- Cloudflare, Markdown for Agents documentation. https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/, read 2026-09.
- Cloudflare, AI Crawl Control documentation. https://developers.cloudflare.com/ai-crawl-control/, read 2026-09.
- Cloudflare, Web Bot Auth documentation. https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/, read 2026-09.
Last reviewed 2026-11-05