The Accidental AI Lockout: 1 in 13 Small Businesses Has Blocked ChatGPT Without Knowing It
We read the public robots.txt and llms.txt files of 335 reachable small-business websites across five industries and fifteen US cities. Some businesses have locked AI assistants out of their websites - and the fingerprint says most did it by accident.
Why we ran this study
While preparing routine site reviews, we found a Sydney dental clinic whose robots.txt blocked GPTBot and ClaudeBot - meaning ChatGPT and Claude cannot read that practice's website at all. Nothing about the site suggested this was deliberate. We wondered how common that is among ordinary small businesses - dentists, lawyers, solar installers, realtors, plumbers - rather than the top-1000 giant sites that existing AI-crawler studies focus on.
What we found
1. The lockouts exist - and they look accidental
10 of the 230 robots.txt-bearing sites (4.3%) fully block at least one major AI crawler. The most-blocked: GPTBot (9 sites), meta-externalagent (9), CCBot (8), ClaudeBot (7), Google-Extended (7), Bytespider (7).
Here is the tell: most locked-out sites block a near-identical list of six bots. That pattern is the fingerprint of a plugin, theme or CDN "block AI bots" default switch - not ten separate business owners making ten deliberate policy decisions. A dentist in Chicago, a dental group in Atlanta, realtors in Charlotte, Phoenix and Columbus - all carrying the same copy-pasted lockout.
What it means in practice: when a patient asks ChatGPT "who does braces near me in Chicago?", a practice blocking GPTBot cannot be read - the model relies on whatever third-party scraps it already has. The business paid for a website that machines are politely declining to enter, because the front door asked them to.
2. Four sites block everything - including Google
4 sites carry a wildcard "Disallow: /" for all user-agents. That is not an AI policy - that is invisibility. These are usually staging-site settings that were never switched off after launch.
3. The surprise: small businesses are adopting llms.txt faster than reported
55 of 335 reachable sites (16.4%) publish a genuine llms.txt file - verified by us as actual markdown-style AI guidance, not a server misconfiguration returning HTML. (We found 15 of those false positives and excluded them.) Law firms lead: 16 of 78. For a convention that barely existed two years ago, one small business in six is remarkable - and far above what studies of big-brand sites report.
4. By industry (fully locked out / sample)
Real estate agencies: 6/128 · Dentists: 2/39 · Solar installers: 2/79 · Law firms: 0/78 · Plumbers: 0/11. Law firms were both the least locked-out and the strongest llms.txt adopters - consistent with an industry that takes written policy seriously.
Methodology - reproduce any line of this yourself
Sample: 540 business websites collected from public YellowPages directory listings across five search categories (dentist, law firm, solar installer, real estate agency, plumber) in fifteen US cities, August 4-5, 2026. We removed directory-hosted subdomains and franchise-platform pages (174), leaving 366 independently-owned sites; 335 were reachable at check time.
Checks: For each site we fetched exactly two public files - /robots.txt and /llms.txt - with a transparently-identified research crawler, one site at a time, seconds apart. No page content was crawled. Reading robots.txt is how every crawler on earth is supposed to behave: it is the public, machine-readable policy file of a website.
Classification: A bot counts as "fully blocked" only when its user-agent group contains "Disallow: /" with no overriding Allow. Partial disallows are recorded separately. llms.txt counts as present only when the response is a genuine text/markdown guidance file, not an HTML fallback page.
Verify it: every flagged site's exact robots.txt rule is in the dataset below - open any domain's /robots.txt in your browser and compare. It takes ten seconds per site.
Download the full raw dataset (JSON, all 335 records + evidence)
Honest limits of this study
This is a snapshot of one sample, five industries, fifteen US cities, on two days in August 2026 - not a census of the small-business web. Percentages describe this sample only. robots.txt files change; a site flagged today may be fixed tomorrow (we would be glad). We do not know any site owner's intent - "accidental" is our inference from the copy-paste pattern, clearly labeled as inference. And blocking AI crawlers is every owner's right; our finding is only that the pattern suggests many don't know they're doing it.
Common questions
Is it wrong for a business to block AI crawlers?
No - it is a legitimate choice, and some publishers make it deliberately to protect content. The concern this study raises is different: the copy-paste pattern suggests many small businesses are blocking AI assistants without having made any choice at all, while simultaneously paying for visibility.
How do I check my own site?
Open yourdomain.com/robots.txt in a browser and look for lines like "User-agent: GPTBot" followed by "Disallow: /". Or run our free Through Machine Eyes tool, which checks this along with nine other machine-visibility signals.
Was this crawl polite and legal?
Yes. We fetched only two public policy files per site - the very files that exist to be read by crawlers - with a transparently identified user-agent linking back to this page, spaced out over time. No page content, no personal data, no bypassing of any block.
Related research: The 20-Agency AEO Readiness Study (linked by Search Engine Roundtable) · The Re-Test, one month later · Our public claims ledger.