Which AI crawlers to allow in robots.txt, and which only collect training data
By Roshan, founder of SearchMaxx - - 3 min read
If ChatGPT, Claude or Perplexity cannot fetch your pages, they cannot cite them. The most common reason we see is a robots.txt rule someone added to block "AI" without knowing which bot does what.
Each company runs separate bots for separate jobs. Blocking the training bot keeps your content out of future models. Blocking the search bot keeps you out of AI search answers today. Most software companies want the first choice to be theirs and the second to be "allow".
The bots that matter
| Company | Bot | What it does | What blocking it does |
|---|---|---|---|
| OpenAI | OAI-SearchBot |
Finds sites to show in ChatGPT search | Your pages stop appearing in ChatGPT search answers |
| OpenAI | GPTBot |
Collects content that may train OpenAI's models | Signals that your content should not be used for training |
| OpenAI | ChatGPT-User |
Visits a page when a ChatGPT user asks about it | robots.txt "may not apply", because a user started the visit |
| Anthropic | Claude-SearchBot |
Indexes pages to improve Claude's search results | Can reduce your visibility in Claude's search answers |
| Anthropic | ClaudeBot |
Collects content that may train Anthropic's models | Excludes your future content from training |
| Anthropic | Claude-User |
Fetches a page when a Claude user's question needs it | Stops those fetches, which can reduce your visibility |
| Perplexity | PerplexityBot |
Finds and links sites in Perplexity's answers | Your pages stop appearing in Perplexity results |
| Perplexity | Perplexity-User |
Visits a page a user asked about | Generally ignores robots.txt, because a user asked |
Googlebot |
Crawls for Google Search, including AI Overviews | Removes you from Google Search entirely | |
Google-Extended |
Not a bot. A robots.txt token for Gemini training and grounding | Does not affect Google Search or ranking |
Sources: OpenAI's crawler docs, Anthropic's crawler policy, Perplexity's crawler docs, and Google's list of crawlers.
Two details catch people out:
- Google AI Overviews come from Googlebot. Google says
Google-Extended"does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." BlockingGoogle-Extendeddoes not take you out of AI Overviews. BlockingGooglebottakes you out of everything. - Blocking a training bot does not touch the search bot. You can block
GPTBotand still allowOAI-SearchBot. They are separate rules.
A robots.txt you can copy
This version lets every AI search bot in and opts out of model training. If you are happy to be used for training too, delete the last three groups.
# AI search: allowed, so assistants can find and cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
# Model training: opted out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
A crawler follows the most specific group that names it, and ignores the * group when a named group exists. If you list a bot by name, give it every rule it needs in that group.
Check that it worked
robots.txt is not the only thing that blocks bots. Firewalls and bot protection often refuse AI crawlers before robots.txt is read, and you will not see it from your own browser. You can test the server's answer yourself:
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; OAI-SearchBot)" https://yoursite.com/
A 200 means the page loaded. A 403 or 503 means something in front of your site is turning the bot away. Some firewalls check the bot's IP address as well as its name, so a 200 here is good but not proof.
Our free check runs both tests for every AI search bot in about 10 seconds.
Want this done for your site?
Every day we make one change to your website that helps AI engines name you, and a person checks each one. Email me your website and the question your buyers ask AI.