Technical AEO for AI Crawlers

Why this comes first

Every other part of AEO — structure, schema, authority, brand monitoring — is downstream of one question: when an AI crawler requests your page, does it get your content?

If the answer is no, everything else is theatre. And the answer is no more often than teams expect, because crawler access is configured once, usually by someone who has left, and never revisited.

This is the highest-yield work in AEO precisely because it is boring. Nobody puts a robots.txt audit in a strategy deck.

Check 1: robots.txt

Start here, and check the live file rather than what you believe it contains.

The crawlers that matter today:

• GPTBot — OpenAI, for training and retrieval
• OAI-SearchBot — OpenAI, for ChatGPT search results
• ChatGPT-User — OpenAI, for user-initiated browsing
• ClaudeBot — Anthropic, for crawling
• Claude-User — Anthropic, user-initiated fetches
• PerplexityBot — Perplexity indexing
• Perplexity-User — Perplexity user-initiated fetches
• Google-Extended — controls Gemini grounding use, separately from Googlebot
• Bingbot — still relevant for Copilot surfaces
• Applebot-Extended — Apple Intelligence

A permissive baseline:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml

Three failure patterns to look for specifically:

The inherited blanket block. A Disallow: / under a wildcard, or an explicit AI-crawler block added during the 2023–24 wave of publisher blocking, still in place with nobody aware.

The partial block. Disallow: /pricing/ or /docs/ added for an unrelated reason years ago. Your most citation-worthy pages are frequently the ones sitting behind these.

The staging leak. A Disallow: / from a staging config deployed to production. Rare, catastrophic, and it happens.

Note that Google-Extended is a distinct control from Googlebot. Blocking it removes you from Gemini grounding while leaving conventional Google search untouched, which is a decision worth making explicitly rather than by accident.

Check 2: server-rendered content

Fetch your own page the way a crawler does and read what comes back:

curl -A "GPTBot" -sL https://example.com/pricing | wc -c

Then look at the actual HTML. The question is whether your substantive content — prices, feature descriptions, comparison details — is present in that response, or whether you received an application shell that fills itself in later.

Client-side rendering is the most common structural cause of AI invisibility on modern sites. A React or Vue application that renders content after hydration is fine for users, fine for Googlebot most of the time, and frequently empty to AI crawlers that do not run JavaScript or that give up early.

The fix is server-side rendering or static generation for any page you want cited. This is a genuine engineering commitment, not a configuration toggle, which is why it is often deferred — and why sites that have done it enjoy a structural advantage over competitors that have not.

Two related traps:

• Content behind interaction. Tabs, accordions and "read more" toggles that fetch content on click. If the text is not in the initial HTML, assume it does not exist.
• Content in images. Pricing tables and comparison charts rendered as images are invisible. Put the numbers in HTML and use the image as decoration.

Check 3: bot mitigation

This is the failure teams find last, because everything looks correct in configuration and still does not work.

WAFs, DDoS protection and bot-management products frequently classify AI crawlers as suspicious automated traffic. The crawler receives a JavaScript challenge, a CAPTCHA, a 403, or a rate-limit response. Your robots.txt says yes; your edge says no.

How to find it:

1. Filter your CDN or WAF logs by AI user agent.
2. Look at the status-code distribution. Successful content responses should dominate.
3. Investigate any meaningful share of 403, 429, 503 or challenge responses.
4. Check whether your bot-management ruleset has a "known bots" allowlist and whether AI crawlers are on it.

Most enterprise edge platforms now ship verified-bot categories that include the major AI crawlers. Enabling that category is usually a five-minute change with a disproportionate effect.

Rate limiting deserves separate attention. AI crawlers can arrive in bursts. A limit tuned for human traffic can throttle a crawl that would otherwise have indexed your whole documentation set.

Check 4: continuous monitoring

The reason to treat this as monitoring rather than an audit: all three problems above can reappear without anyone touching AEO. A new WAF rule, a framework upgrade that shifts rendering, an SEO contractor tightening robots.txt, a CDN default changing.

What to watch:

• Crawler serve rate per crawler: proportion of requests answered with real content.
• Coverage: which sections are being crawled, and which are not being visited at all.
• Response times to crawler requests, since slow responses truncate crawls.
• Status-code distribution per user agent, alerting on any shift.

This data lives in your logs. Content-focused AEO tools cannot see it, which is why crawler analytics is a meaningful differentiator when evaluating platforms — several report only on answers, leaving the technical half of AEO unmeasured. Our comparison matrix marks which tools include crawler analytics.

The agent layer

One newer consideration that sits beyond crawling.

An increasing share of non-human traffic is agents attempting tasks rather than crawlers building an index: find the price, check whether a feature exists, start a trial. Agents fail on things crawlers do not care about and human testers never notice — a price revealed on hover, a plan comparison that requires a dropdown selection, a signup flow with a step that silently depends on a third-party script.

When an agent stalls, the user is told your product is unclear or unavailable. You will never see this in analytics, because there was no conversion and no complaint.

Testing for it means having an agent actually attempt the tasks. This is what CiteCue's Agent Usability does — it runs real task attempts and reports the step where the agent stopped — and it is the rarest capability in the category. Very few competitors document anything equivalent, which makes it the clearest structural gap in most AEO stacks today.

A one-hour audit

In order, because each step gates the next:

1. Read your live robots.txt. Confirm the AI crawlers you want are allowed.
2. curl your three highest-value pages with an AI user agent. Confirm the substance is in the HTML.
3. Filter CDN logs by AI user agent. Check the status-code mix.
4. Confirm AI crawlers are allowlisted in bot management.
5. Check your rate limits against observed crawler burst behaviour.
6. Set an alert on crawler serve rate so regressions surface without a human remembering to look.

Steps 1 to 3 take about twenty minutes and find most problems. Skipping to content optimisation before completing them is the most common way AEO programmes waste a quarter.

Further reading

llms.txt Explained covers the discovery file and why it is not a substitute for access. Why Your Brand Is Not Being Cited puts these checks into a full diagnostic sequence.