Reference
Every entry below is taken from the operator’s own documentation. For each AI crawler you get the exact user agent, who runs it, whether it feeds model training or AI search, what you lose by blocking it, and how to confirm it is the real bot and not a spoof.
Maintained by Synthesis Insights. Facts verified 3 September 2026 against each operator’s published crawler documentation.
The one distinction that matters before you block anything
Training crawlers and AI-search crawlers are different jobs. Blocking a training crawler (GPTBot, ClaudeBot, Amazonbot) or a training-control token (Google-Extended, Applebot-Extended) opts you out of AI training and changes nothing about whether an assistant recommends you. Blocking an AI-search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) is what actually removes you from the answer. Most sites want to block the first group and keep the second. The table makes the difference explicit for every agent.
Sorted by operator. “Verify” is how you confirm a request is genuinely from that bot rather than something spoofing its user agent.
| User agent | Operator | Type | What it feeds | Block it and you lose | Verify it is genuine |
|---|---|---|---|---|---|
GPTBot | OpenAI | Training | Training data for OpenAI foundation models | Opts you out of training use. Does not affect whether ChatGPT search cites you. | openai.com/gptbot.json |
OAI-SearchBot | OpenAI | AI search | Websites shown in ChatGPT search results | You stop being eligible to appear in ChatGPT search. Usually the one to keep. | openai.com/searchbot.json |
ChatGPT-User | OpenAI | User fetch | Pages ChatGPT opens when a user asks it to | Users who ask ChatGPT to visit your page get blocked. Not automated crawling. | openai.com/chatgpt-user.json |
ClaudeBot | Anthropic | Training | Training data for Claude models | Opts you out of training. Does not affect whether Claude search cites you. | claude.com/crawling/bots.json |
Claude-SearchBot | Anthropic | AI search | Search results inside Claude | You stop being eligible for Claude search citations. | claude.com/crawling/bots.json |
Claude-User | Anthropic | User fetch | Pages Claude opens when a user asks it to | Users who ask Claude to visit your page get blocked. | claude.com/crawling/bots.json |
PerplexityBot | Perplexity | AI search | Pages surfaced and linked in Perplexity answers | You stop being eligible for Perplexity citations. Not used for model training. | perplexity.com/perplexitybot.json |
Perplexity-User | Perplexity | User fetch | Pages Perplexity opens for a user's question | robots.txt is generally ignored here, since it is a user request. Block by IP if you must. | perplexity.com/perplexity-user.json |
Googlebot | AI search | Google Search, and grounding for AI Overviews and AI Mode | Removes you from Google entirely. Almost never the right move. | Reverse DNS to *.googlebot.com, plus Google’s published crawler IP list | |
Google-Extended | Control token | Whether Google-crawled content trains Gemini and grounds it | No effect on Google Search inclusion or ranking. It is a training-use switch only. | Control token, no separate crawler and no IP list | |
Bingbot | Microsoft | AI search | Bing Search, and the index Microsoft Copilot draws on | Removes your Bing presence and your Copilot grounding. | Reverse DNS to *.search.msn.com (Bing Webmaster Tools verification) |
Amazonbot | Amazon | Training | Amazon products and services; may train Amazon AI models | Opts you out of Amazon’s crawl and AI training use. | developer.amazon.com/amazonbot/ip-addresses |
Applebot | Apple | AI search | Spotlight, Siri, and Safari suggestions | Removes you from Apple’s search surfaces. | Reverse DNS to Apple’s crawler range |
Applebot-Extended | Apple | Control token | Whether Apple-crawled content trains Apple’s models | No effect on Spotlight, Siri, or Safari inclusion. Training-use switch only. | Control token, no separate crawler and no IP list |
meta-externalagent | Meta | Training | Training Meta’s foundation models, and indexing content | Opts you out of Meta’s AI training crawl. | robots.txt; changes take up to 24 hours |
meta-externalfetcher | Meta | User fetch | Links Meta AI fetches at a user’s request | May bypass robots.txt, since it is a user-initiated fetch. | robots.txt / IP |
One correction worth making, because stale guides still repeat it: anthropic-ai is not an active Anthropic crawler. Anthropic’s current agents are the three ClaudeBot entries above.
A user agent string is one line of text and anyone can send it. A scraper can call itself GPTBot to slip past a rule that trusts the name. Two checks defeat that, and every serious operator supports at least one:
*.googlebot.com, and Bingbot to *.search.msn.com. Confirm it forward-resolves back to the same IP.These come up in the same log lines but have thinner public documentation. Treat the operator and purpose below as directional, not publisher-confirmed to the standard of the table above.
| User agent | Operator | What it is | |||
|---|---|---|---|---|---|
Bytespider | ByteDance (TikTok) | Training crawler. Historically aggressive, and compliance with robots.txt has been inconsistent. No publisher IP-verification endpoint. | |||
CCBot | Common Crawl | Training | The free Common Crawl public dataset on Amazon S3, a corpus many AI labs and researchers train on | Removal from an open research corpus, not from any one company’s model. Common Crawl also runs a separate opt-out registry. | index.commoncrawl.org/ccbot.json, plus reverse DNS to crawl.commoncrawl.org |
cohere-ai | Cohere | Associated with training-data collection. Thin public documentation. | |||
facebookexternalhit | Meta | Link-preview crawler for shared posts (title, description, thumbnail). Not AI training. Listed here because it is often mistaken for one of the AI agents above. |
Common questions
It is the crawler for a third-party SEO audit tool, not an AI assistant. Seeing it in your logs means someone ran an SEO scan of your site. It has nothing to do with ChatGPT, Claude, Perplexity or Gemini, and it does not affect whether AI assistants cite you.
Read your robots.txt against the table above, then check your server access logs for these user agent strings. robots.txt tells you what you have allowed or blocked; the logs tell you who actually came. The reference posts linked below walk through both.
No. GPTBot only collects training data. Whether ChatGPT cites you in its search answers is decided by OAI-SearchBot, a separate agent. You can block GPTBot and still be recommended in ChatGPT search, as long as OAI-SearchBot is allowed.
Allow it in robots.txt (User-agent: OAI-SearchBot, then Allow: /) and make sure nothing at the firewall or CDN is blocking the IP ranges OpenAI publishes at openai.com/searchbot.json. Being reachable by OAI-SearchBot is what makes you eligible for ChatGPT search citations.
A user agent string is trivial to spoof, so never trust it alone. Match the request’s source IP against the operator’s published IP list (linked in the table), or run a reverse DNS lookup where the operator supports one. If neither matches, it is not the real bot.
No, and this is the distinction that matters most. Training crawlers (GPTBot, ClaudeBot, Amazonbot, meta-externalagent) and the training-control tokens (Google-Extended, Applebot-Extended) do not decide whether you are cited. The AI-search agents (OAI-SearchBot, PerplexityBot, Claude-SearchBot) do. Block the first group and keep the second, and your citation eligibility is intact.
These walk through the checks the table points at, on your own logs and robots.txt:
Want the whole thing checked for you?
The free AI Agent Readiness Check tests whether an AI agent can actually reach and use your site, and reports the specific elements at fault. It takes about a minute and does not need a call.