# The robots.txt lines that quietly block AI crawlers

> Most blocked sites were blocked by accident, and the file still looks fine at a glance.

robots.txt is the first thing a crawler reads and the last thing an owner checks. It is plain text, it is
    easy to copy from a template, and a single line can remove a site from every AI answer while the site keeps
    loading perfectly for humans.

## The three lines that do the damage

### 1. The wildcard block

`User-agent: *` followed by `Disallow: /`. This is the big one. It reads like
    "no crawlers", but it also catches the AI crawlers, and it is often pasted in by a template, a staging
    setup that was never reverted, or a security plugin with a switch labelled "block bots".

### 2. The named block

`User-agent: GPTBot` / `Disallow: /` — and the same for `ClaudeBot`,
    `PerplexityBot`, `OAI-SearchBot`, `Google-Extended`. Sometimes this is a
    deliberate decision ("we do not want our content used for training") that nobody remembers making.

### 3. The rule that is not in robots.txt at all

A firewall, a CDN bot rule, or a security plugin returning `403` to anything with "bot" in the
    user-agent. Nothing in your file is wrong; the request never reaches the page. You cannot see this by reading
    the file — only by making the request as that crawler.

## Once blocked, what actually happens

      - The assistant does not read your page, so it cannot quote it, compare it, or verify your prices.

      - It answers from other sources — usually competitors who never blocked anything.

      - Nothing breaks for humans. Your site looks and performs exactly the same, which is why this sits
      undetected for months.

## Do you actually want to block training?

There is a real distinction worth deciding deliberately, because it is not all-or-nothing:

      - **Training crawlers** (GPTBot, ClaudeBot, Google-Extended) collect pages for model training.

      - **Answer-time fetchers** (OAI-SearchBot, ChatGPT-User, PerplexityBot) fetch a page to answer a
      live question — the moment a customer is looking for you.

Blocking the first while allowing the second is a legitimate position for many publishers. Blocking both
    means you are invisible at the moment of intent, which is the opposite of what most small businesses want.

## The fix, in order

      - Fetch `/robots.txt` and find every block that could match a crawler.

      - Decide per crawler, not in bulk — then change the rule, not the plugin.

      - Re-request the page as that user-agent to confirm it now returns 200.

      - Do it again after any hosting, firewall or plugin change, because this is the kind of setting that
      comes back.

HTML version: https://websitespa.net/guides-robots-txt-ai-crawlers