AI SEO Checklist: 8 Checks AI Agents Run on Your Site

AI agents are the web's newest audience, and most sites fail them in the same handful of ways. This is the checklist we score the 1219 most-visited sites on the web against every week — the same eight checks, in the same order, with a way to test each one yourself in about a minute. The panel's average right now is 50/100.

What an agent actually needs

Not a chatbot, not an API key, not a partnership. A production AI agent working on someone's behalf makes plain HTTP requests and reads what comes back. It cannot run your JavaScript, solve your bot challenge, or click through a cookie wall. If the answer to "what does this cost?" only exists after a script executes, the agent never sees it — and in our daily Field Test, that is the single most common reason an errand fails.

The checklist

A1 · Publish a substantive llms.txt 15 points

The llms.txt convention is a plain-text file at your site root that tells AI readers where the important things are. "Substantive" means it carries real links — docs, pricing, support, legal — not a paragraph of marketing copy. 13% of the panel publishes one.

• Serve it at /llms.txt with Content-Type: text/plain.
• One H1 (your name), a one-line summary, then H2 sections of link lists with a short note per link.
• Link the pages an assistant would need to finish a task: pricing, cancellation, contact, API docs, terms.

curl -s https://yourdomain.com/llms.txt | head -20

See llms.txt and OpenAPI for AI agents for a full template.

A2 · Decide your AI-crawler policy on purpose 15 points

Your robots.txt is read by GPTBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and CCBot. Blocking all of them is a legitimate stance — we label it "closed by policy" and report it as one — but do it knowingly. Two things to know: Claude-User and similar agents fetch on behalf of a person who asked, which is different from a training crawler; and publishing llms.txt while blocking every AI crawler is a contradiction we flag as the paradox. 26% of the panel blocks at least one crawler; 18% blocks them all.

curl -s https://yourdomain.com/robots.txt | grep -iA2 'gptbot\|claudebot\|claude-user\|perplexitybot\|google-extended\|ccbot'

A3 · Make the homepage readable without a browser 25 points

The heaviest-weighted check, because it is the one that ends the most errands. A plain fetch of your homepage should return the actual content as HTML text. A JavaScript shell with an empty <div id="root"> fails. A Cloudflare "checking your browser" interstitial fails. Server-render or pre-render the pages that carry answers.

curl -sL https://yourdomain.com/ | sed 's/<[^>]*>/ /g' | tr -s ' \n' | wc -w
# under ~150 words of visible text, or a page titled 'Just a moment', is a fail

A4 · Add schema.org JSON-LD 15 points

A valid <script type="application/ld+json"> block on the homepage — Organization or WebSite at minimum, Product with Offer on pricing pages, FAQPage where you answer questions. Agents parse it directly; it is the cheapest structured signal you can ship.

curl -sL https://yourdomain.com/ | grep -o '<script type="application/ld+json">[^<]*' | head -c 400

A5 · Make the sitemap discoverable 10 points

Either a Sitemap: line in robots.txt or a file at /sitemap.xml. Include <lastmod> — it is how crawlers know what changed without re-fetching everything.

curl -s https://yourdomain.com/sitemap.xml | grep -c '<loc>'

A6 · Link pricing, support, docs and legal from the homepage 20 points

Every everyday errand starts from the front door: what does it cost, how do I cancel, how do I reach a person, what does the policy say. Those pages need to be reachable by following an <a href> in the homepage HTML — not a menu that only exists after JavaScript runs, and not behind a login.

curl -sL https://yourdomain.com/ | grep -oi 'href="[^"]*\(pricing\|support\|help\|docs\|terms\|privacy\)[^"]*"' | sort -u

B1 · Advertise an MCP server +5 bonus

If you expose tools to agents over the Model Context Protocol, say so at /.well-known/mcp.json. Most sites skip this; the ones that ship it are the ones agents can actually act on rather than just read.

B2 · Publish an OpenAPI document +5 bonus

A reachable OpenAPI description at a stable URL, linked from your docs and your llms.txt. Only relevant if you have an API — but if you do, it is how an agent learns to call it correctly the first time.

What fails most, in practice

Every week a real agent attempts ten errands on real sites and we publish the transcript. The recurring killers, in order: pricing pages that render only in JavaScript; help centers behind a bot challenge; cancellation flows that require a login to even describe; and "contact us" pages with no contact on them. In the latest episode the agent finished 9 of 10 errands across 94 page reads and hit 6 bot walls.

A three-day plan

• Day 1 — ship llms.txt, fix robots.txt to say what you mean, add JSON-LD to the homepage.
• Day 2 — server-render the pricing, support and legal pages; link all four from the homepage HTML.
• Day 3 — publish a sitemap with lastmod; if you have an API, publish OpenAPI and link it.