A Major AI Company Has No robots.txt. We Found Out by Accident.
Our radar watches the internet's rulebooks so owners don't have to check them by hand. This week it pointed at a rulebook that doesn't exist — and the only witness to the mistake was the watcher itself.
The alert that woke me up
At 9:01 on Friday morning, our Policy Radar mailed me a priority alert: “Qwen robots.txt changed.” Second change in four days, it said.
I'll be honest about what I felt: excitement. A major AI company editing its crawler policy twice in one week? That's the kind of thing we live for here. I asked the team to pull the diff so we could see what exactly Qwen changed.
That's when the morning got interesting.
The file that wasn't
The “robots.txt” our radar had been watching weighed 94 kilobytes. If you've ever seen a robots.txt, you know they weigh a few hundred bytes — they're little text files, a handful of rules. 94 KB is not a rules file. 94 KB is a webpage.
And that's what it was. qwen.ai/robots.txt doesn't serve a robots file — it redirects you to the marketing homepage. Our radar had spent days faithfully reporting changes to... their homepage banner, basically. The “policy changes” were a sales page breathing.
So before anything else: we got fooled. Our own tool logged a file that doesn't exist, and we only caught it because we went to read the actual bytes. We corrected the radar's label the same day — the fix is visible on our public radar page. I'd rather tell you we were wrong for three days than pretend we're never wrong at all.
A policy file exists only if the response is a text file that parses as one. Everything else is a homepage in a costume.
Scene illustration, AI-generated. It depicts no real person, client, place or event.
Now the part that actually matters
Once the embarrassment faded, the real finding sat there staring at us: one of the world's major AI companies has no robots.txt on its main domain.
Is that against the rules? No — there are no rules. robots.txt was never law, it's a handshake. No file simply means “crawl everything, everyone, welcome.”
But think about the irony for a second, because it's not small. AI companies ask the entire web to maintain careful robots.txt files — to welcome or refuse their crawlers by name, to keep crawler policy tidy so their machines know where they stand. We've all been told to be machine-readable. And here's a front door of the AI world itself, answering the most machine-read URL on any website with... a redirect to a sales page.
Small businesses get this wrong by accident and nobody tells them — we measured exactly that in our Accidental AI Lockout study this August: 335 small-business sites, raw data public, 1 in 13 accidentally blocking crawlers they need. Giants skip the file entirely and nobody notices either. The web's machine layer is held together by habit and hope, at every size.
The lesson I'd tattoo on every SEO study
Here's the part that goes beyond Qwen, and it's the reason I'm writing this up instead of just fixing our radar quietly.
Our radar believed the file existed because the server answered “200 OK.” Status codes don't lie, but they leave things out: a redirect-to-homepage returns a perfectly healthy 200 — for the wrong document. Any tool, crawler or research study that counts status codes without reading content will happily record files that aren't there.
This is now the second time we've caught this trap in our own work:
- In our llms.txt research, more than one in five files that “existed” (HTTP 200) turned out to be HTML fallback pages wearing a text file's name. We threw them out. Studies that don't read the bytes are overcounting adoption — quietly, everywhere.
- This week our own radar fell into the identical hole from the other side.
So here's the rule, free to steal: a policy file exists only if the response is a text file that parses as one. Everything else is a homepage in a costume.
Check your own front door — two minutes, no tools needed
- Type yourdomain.com/robots.txt in a browser. You should see plain boring text: User-agent, Disallow, a sitemap line. If you see your homepage, a redirect, anything with your logo on it — your crawler policy doesn't exist, whatever your SEO plugin claims.
- Same test for yourdomain.com/llms.txt if you have one.
- Or let our free Through Machine Eyes tool run this plus nine other checks — it reads the bytes, never just the status codes. Now you know why we built it that way.
The honest footnotes
This is a snapshot of one domain root on October 3, 2026 — anyone can verify it today; Qwen may fix it tomorrow and I'd genuinely applaud. It says nothing about Alibaba's wider crawler setup; subdomains can carry their own files. And those “two changes in four days” were real changes — of the homepage our radar was accidentally watching. Even our mistake was telling the truth about something.
— Manas
Everything here is checkable in one browser tab
qwen.ai/robots.txt for the finding · our public Policy Radar for the corrected row · the Accidental AI Lockout study for the data (raw JSON, CC-BY). We publish our own mistakes because that's the only way you can trust our findings.
Questions people actually ask
Does qwen.ai have a robots.txt?
No — as of October 3, 2026 the URL qwen.ai/robots.txt redirects to the homepage. Verifiable in any browser, right now.
Is a missing robots.txt a violation?
No — robots.txt is a convention, not a law. Its absence simply means “crawl everything, everyone, welcome.”
Why do robots.txt and llms.txt studies overcount?
Because redirects and HTML fallback pages return HTTP 200 and get counted as files unless the content is actually read. A policy file exists only if the response is a text file that parses as one.
Useful next steps
Public policy radar · Lockout study (raw data) · Through Machine Eyes · More field notes · Contact