AI crawlers and robots.txt: which bots to allow

Allow the search and live-fetch crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) if you want to appear inside AI answers. Blocking them removes you from the index those answers are drawn from. Training crawlers such as GPTBot, ClaudeBot and CCBot are a separate decision: blocking them does not cost you answer visibility.

What are AI crawlers?

AI crawlers are the bots that AI products send to your site. They are not one thing, and treating them as one thing is where most robots.txt files go wrong. Each vendor runs several, and they do different jobs:

• Training crawlers collect pages that may be used to train future models. OpenAI documents GPTBot for this, Anthropic documents ClaudeBot, and Common Crawl's CCBot feeds an open dataset many labs use.
• Search crawlers build the index an AI product searches when a user asks a question. OpenAI's OAI-SearchBot and Anthropic's Claude-SearchBot are in this group, as is PerplexityBot.
• Live-fetch agents fetch a page in real time because a user asked about it right now. ChatGPT-User, Claude-User and Perplexity-User behave this way.

The distinction matters because the three groups have different consequences. A training block is a licensing stance. A search or live-fetch block is a visibility decision, and it is usually made by accident.

Which crawlers should you allow?

If your goal is for AI answers to mention and cite you, the minimum policy is: allow every search and live-fetch crawler, then decide on training separately.

• Allow: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User.
• Optional: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai. These are the training and dataset crawlers.

Google-Extended and Applebot-Extended are worth calling out because they are training-only opt-outs. Blocking Google-Extended does not remove you from Google Search, and blocking Applebot-Extended does not remove you from Siri or Spotlight, since those use the regular Googlebot and Applebot instead.

How do you block AI crawlers correctly?

robots.txt groups are matched by user-agent name, and a crawler obeys the most specific group that names it. Two rules follow from that:

• A crawler with its own group ignores the wildcard group entirely. If you write a permissive User-agent: * block and then a User-agent: GPTBot group with Disallow: /, GPTBot is blocked and nothing else changes.
• A crawler with no group of its own falls back to User-agent: *. This is the common accident: a restrictive wildcard rule silently blocks every AI search crawler you never mentioned.

A working policy looks like this:

User-agent: *
Allow: /

User-agent: OAI-SearchBot Allow: /

User-agent: GPTBot Disallow: / ```

Two more mechanics people miss. robots.txt is per host and per scheme, so the file on https://example.com says nothing about https://docs.example.com. And it is advice, not access control: reputable crawlers honor it, but anything genuinely private needs authentication.

Does blocking training crawlers hurt AI visibility?

No, as long as the search and live-fetch crawlers stay allowed. Those are the ones that decide whether your page can be retrieved and cited when someone asks a question today. Training data influences what a model knows in the abstract, which is a much weaker and much slower path to being mentioned than being in the index the product searches at answer time.

The reverse mistake is more expensive. Sites that blanket-block anything with "bot" or "AI" in the name often remove themselves from AI search results while keeping their training exposure roughly unchanged, because plenty of dataset crawling happens through Common Crawl regardless.

How do you check what your robots.txt actually does?

Read the file the way a crawler does, not the way you wrote it. Our AI crawler checker fetches or accepts your robots.txt, evaluates all sixteen crawlers in the registry, and shows the exact rule that decided each verdict, so a wildcard block is visible instead of implied. The robots.txt generator goes the other direction and writes a file with explicit per-crawler groups.

After a change, expect a delay. Each crawler re-reads robots.txt on its own schedule, and the AI search index behind it updates on its own crawl cadence after that.

Frequently asked questions

Does robots.txt stop AI companies from using my content?

It stops the crawlers that honor it, which includes the documented bots from OpenAI, Anthropic, Perplexity and Google. It does not stop content that was already collected, content reachable through a third party such as Common Crawl, or crawlers that ignore the convention. For real enforcement you need authentication or terms that you can act on.

Should I add a noindex tag as well?

Only if you also want out of regular search. A noindex directive removes the page from search engines generally, which is a much bigger decision than an AI crawler policy. Use robots.txt groups when the goal is specific to AI bots.

What about llms.txt, does it control crawlers?

No. llms.txt is a curated description of your site for agents that choose to read it, not a permission file. Crawler permissions live in robots.txt. The two solve different problems, and publishing llms.txt does nothing for access control.

How often should I audit the crawler list?

Vendors add and rename bots, so check when you make robots.txt changes and at least quarterly. New search crawlers are the risky category: a crawler that did not exist when you wrote your file falls into whatever your wildcard group says.