Technical SEO for AI Discovery

Classic technical SEO asks whether a crawler can find, fetch and understand your pages. AI discovery asks the same question of a different reader — one that arrives with a task, reads a page or two, and either finishes the errand or gives up. The fundamentals overlap almost entirely. The failure modes don't.

Where agents fail that crawlers don't

Googlebot renders JavaScript. Most production agents don't — they make a plain HTTP request and read the HTML that comes back. Googlebot is allow-listed through every bot-protection product. Agents are the traffic those products are tuned to stop. And Googlebot has all the time in the world; an agent has a step budget and a person waiting.

So a site can be technically flawless for search and still lose every agent at the door. In our latest Field Test episode, 2 of 141 page fetches hit a bot wall. 7% of the 113 sites on our Index block at least one AI crawler by policy.

1. Render the answer in HTML

Pricing, plan comparisons, cancellation steps, contact details: these must exist in the served HTML, not appear after a script runs. Server-side rendering, static generation, or pre-rendering for the handful of pages that carry answers. The test is one command:

curl -sL https://yourdomain.com/pricing | grep -o '\$[0-9]*' | head

No prices in the output means no prices for an agent. This single check is worth 25 of the 100 points on the Index.

2. Don't wall the readers you want

Bot-protection defaults treat every non-browser request as hostile. That includes an agent fetching your help center because a customer asked how to cancel. Review your rules: distinguish scrapers from named, well-behaved agents; serve the plain HTML version to anything that identifies itself honestly; never put your support and legal pages behind a challenge. And write a robots.txt that says what you mean — blocking everything is a choice, and we report it as "closed by policy" rather than as a bug.

3. Ship structured data agents can use

JSON-LD is read directly, no parsing heuristics needed. Organization and WebSite on the homepage; Product with Offer on every plan; FAQPage wherever you answer a question; HowTo for cancellation and setup flows. Keep it in sync with the visible page — an agent that reads both and finds them disagreeing trusts neither.

4. One hop from the homepage to the money pages

Pricing, support, docs, legal. Each linked from the homepage with a real <a href> in the HTML, not a JavaScript menu, not an icon with no text. An agent that has to guess URLs burns its step budget guessing.

5. llms.txt as the map

A curated, link-rich llms.txt at the root is the fastest way to tell an AI reader where things are. It costs ten minutes and it's checked on every audit. Here is the template.

6. Sitemaps with lastmod

The standard advice, still right: /sitemap.xml or a Sitemap: line in robots.txt, every canonical URL, and a real <lastmod> on each. Crawlers of every kind use it to spend their budget on what changed.

7. Measure it the way it's actually experienced

Fetch your own pages the way an agent does and read what comes back:

curl -sL -A "Mozilla/5.0 (compatible; ClaudeBot/1.0)" https://yourdomain.com/ | head -c 2000

If you see "Just a moment", "Verifying you are human", or an empty app shell, that is what every agent sees. Then look at your site's Index report — each check maps to a fix — and read a few Field Test transcripts to watch a real agent hit real walls, verbatim.