AI Crawler Forensics Audit

Why this became a real problem in 2026

The crawler world split. OpenAI now runs three separate bots with three separate jobs — training, search indexing, and live user fetches. Anthropic runs four. Blocking the wrong one means your content trains someone's model but never gets cited; blocking another means you vanish from AI search results entirely while your competitor's pricing page gets read aloud to your prospects. Most sites made their robots.txt decisions in 2023 and never looked again.

What the forensics covers

• The full crawler registry check: your site tested against 28 documented AI user agents — who gets in, who gets a 403, who gets silently served a challenge page by your CDN (the failure nobody sees).
• Intent vs. reality: we ask what you actually want — train AI models, appear in AI search, both, neither — then measure the gap between that and your current configuration.
• The three silent failure modes: CDN bot protection blocking crawlers your robots.txt allows; wildcard rules overriding specific ones; and crawler tokens that changed names while your config kept the old ones.
• Log evidence where available: if you can export server logs, we show you which AI crawlers actually visited in the last 90 days — often a shorter list than owners expect.
• The fix, written out: a corrected robots.txt and CDN rule set, explained line by line, ready to deploy — plus verification steps so you can confirm each crawler's access yourself.

What this is not

• Not a promise of AI citations — access is the precondition, not the guarantee. Anyone who guarantees citations is selling weather control.
• Not a generic SEO audit with "AI" in the title. This is specifically about machine access: who reads you, who can't.