AI crawler checker
Paste your robots.txt (or fetch it by domain) and see, crawler by crawler, whether ChatGPT, Claude, Perplexity and the other AI bots can read your site. Every verdict shows the exact rule that decided it.
How to use
- Paste your robots.txt content, or switch to Fetch by domain and enter your domain to pull the live file.
- Run the check and review the table: each of the 16 known AI crawlers shows allowed or blocked, its category (Training, Search or Live-fetch), and the rule that matched.
- If search or live-fetch crawlers are blocked, copy the allow snippet and paste it into your robots.txt.
- Re-run the check to confirm every Search and Live-fetch crawler shows allowed.
Frequently asked questions
Which AI crawlers does the checker cover?
Sixteen crawlers from OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended), Apple (Applebot-Extended), Common Crawl (CCBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider), Amazon (Amazonbot) and Cohere (cohere-ai).
What is the difference between Training, Search and Live-fetch crawlers?
Training crawlers collect data to train models. Search crawlers index your site so AI answers can cite it. Live-fetch crawlers read pages in real time when a user asks about you. Blocking training bots does not keep you out of AI answers, but blocking search and live-fetch bots does.
Should I block AI training crawlers?
That is a policy choice. If you want your content quoted inside ChatGPT, Claude and Perplexity answers, you must allow their Search and Live-fetch crawlers. You can block Training crawlers at the same time without losing answer visibility.
Why does the checker say a crawler is blocked when I never mentioned it?
A wildcard group (User-agent: *) with Disallow: / blocks every crawler that has no explicit group. The checker shows the exact rule that matched, so you can see when a wildcard rule is the cause.
Does changing robots.txt update AI answers immediately?
No. Each crawler re-reads robots.txt on its own schedule, from hours to weeks. AI search indexes then update on their own crawl cadence after that.