robots.txt is the first thing a crawler reads and the last thing an owner checks. It is plain text, it is easy to copy from a template, and a single line can remove a site from every AI answer while the site keeps loading perfectly for humans.
The three lines that do the damage
1. The wildcard block
User-agent: * followed by Disallow: /. This is the big one. It reads like
"no crawlers", but it also catches the AI crawlers, and it is often pasted in by a template, a staging
setup that was never reverted, or a security plugin with a switch labelled "block bots".
2. The named block
User-agent: GPTBot / Disallow: / — and the same for ClaudeBot,
PerplexityBot, OAI-SearchBot, Google-Extended. Sometimes this is a
deliberate decision ("we do not want our content used for training") that nobody remembers making.
3. The rule that is not in robots.txt at all
A firewall, a CDN bot rule, or a security plugin returning 403 to anything with "bot" in the
user-agent. Nothing in your file is wrong; the request never reaches the page. You cannot see this by reading
the file — only by making the request as that crawler.
Once blocked, what actually happens
- The assistant does not read your page, so it cannot quote it, compare it, or verify your prices.
- It answers from other sources — usually competitors who never blocked anything.
- Nothing breaks for humans. Your site looks and performs exactly the same, which is why this sits undetected for months.
Do you actually want to block training?
There is a real distinction worth deciding deliberately, because it is not all-or-nothing:
- Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect pages for model training.
- Answer-time fetchers (OAI-SearchBot, ChatGPT-User, PerplexityBot) fetch a page to answer a live question — the moment a customer is looking for you.
Blocking the first while allowing the second is a legitimate position for many publishers. Blocking both means you are invisible at the moment of intent, which is the opposite of what most small businesses want.
The fix, in order
- Fetch
/robots.txtand find every block that could match a crawler. - Decide per crawler, not in bulk — then change the rule, not the plugin.
- Re-request the page as that user-agent to confirm it now returns 200.
- Do it again after any hosting, firewall or plugin change, because this is the kind of setting that comes back.