An AI crawler is an automated program run by an AI company that fetches web pages for one of three jobs: collecting content that may be used to train its models, building the search index its assistant draws sources from, or fetching a page live because a user asked a question. OpenAI, Anthropic and Perplexity each document separate user agents for these jobs, so a site can treat training and search differently in robots.txt.
Which crawler does which job
| Company | Training | Search index | Live fetch for a user |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed | PerplexityBot | Perplexity-User |
Perplexity says neither of its agents is used to collect content for training AI foundation models.
Google works differently. Google-Extended is a robots.txt token with no user agent of its own. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini apps and Vertex AI, and Google says it has no effect on inclusion or ranking in Google Search. AI Overviews and AI Mode retrieve pages from the regular Search index, so ordinary Search crawling is what matters for them.
Why AI crawlers matter for SEO and GEO
An engine can only cite what its systems can fetch. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, apart from navigational links. Anthropic says blocking Claude-SearchBot may reduce a site's visibility in its search results, and Perplexity recommends allowing PerplexityBot for sites that want to appear in its results.
Rendering matters too. In a December 2024 study of crawler traffic on Vercel's network, Vercel and MERJ found that none of the major AI crawlers they tracked rendered JavaScript, including all three of OpenAI's and Anthropic's ClaudeBot. Google's Gemini, which uses Googlebot's infrastructure, could. Text that only appears after scripts run can be missing from what those crawlers see; see why server-rendered HTML matters and our crawler test of nine pricing pages.
An example: allow search, refuse training
OpenAI's documentation says each of its settings is independent, and describes this case: allow OAI-SearchBot so the site can appear in search results, and disallow GPTBot to signal that its content should not be used to train OpenAI's foundation models. In robots.txt:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
OpenAI says its search systems can take about 24 hours to reflect a robots.txt change, and Perplexity says up to 24 hours. How to set up robots.txt for AI crawlers covers every engine.
Common misconceptions
- "Blocking GPTBot keeps a site out of ChatGPT." GPTBot is the training crawler. Appearing in ChatGPT search depends on OAI-SearchBot.
- "Every AI fetch obeys robots.txt." OpenAI says robots.txt rules may not apply to ChatGPT-User because a user starts those actions, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.
- "Google-Extended controls AI Overviews." It does not affect Google Search. The switch for AI Overviews and AI Mode is the Search generative AI control in Search Console.
- "Blocking IP ranges is a cleaner opt-out." Anthropic warns that IP blocking may not work reliably, because it stops its bots from reading robots.txt.