How to set up robots.txt for AI crawlers

Allow the crawlers that feed AI search, and treat the training crawlers as a separate decision. OpenAI, Anthropic, Perplexity and Google each document different names for these jobs, so one robots.txt file can keep your pages citable in ChatGPT, Claude and Perplexity while opting out of model training.

It is easy to get wrong in both directions. One Disallow: / under the wrong name can take you out of an engine's answers, and rules written under User-agent: * do not reach any crawler you name in a group of its own. If the term is new, start with what an AI crawler is.

The crawler names and what each one controls

For citations, the names that matter are OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot, plus Googlebot for Google's AI features. GPTBot and ClaudeBot are about training. OpenAI states that its settings are independent, so a site can appear in ChatGPT search while opting out of training.

Google-Extended needs a second look. Google says it covers grounding in Gemini Apps as well as training, so blocking it is not a training-only choice.

How crawlers read the file

The standard is RFC 9309, published by the IETF in September 2022. These are the rules that cause most mistakes:

• Location. The file must be at /robots.txt in the top level of the host, as UTF-8 plain text. Google applies it only to the host, protocol and port it is served from, and Anthropic asks site owners to repeat an opt-out on every subdomain.
• Groups. A group is one or more User-agent lines followed by rules. Names are matched without regard to case.
• Named groups win. A crawler follows the group that names it and falls back to the * group only when no group names it. A crawler you name gets none of your * rules.
• Longest match. The most specific matching path wins. When an Allow and a Disallow are equally specific, the RFC says Allow should be used.
• Errors. If robots.txt returns a 4xx error, crawlers may fetch anything. If it returns a 5xx error, they must treat the whole site as disallowed.
• Caching. Crawlers should not use a cached copy for more than 24 hours. OpenAI says its search systems take about 24 hours to adjust, and Perplexity says up to 24 hours.
• Crawl-delay. It is not part of the standard. Anthropic honors it; Google's documentation lists it as unsupported.

A starting file for a site that wants to be cited

Adapt the paths to your site. The example keeps an admin area and a checkout private, lets the AI search crawlers in, and opts out of OpenAI and Anthropic training.

# AI search and answer crawlers: allowed, same private paths closed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /checkout/

# Model training: opted out (delete this group to allow training)
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Everyone else, Googlebot included
User-agent: *
Disallow: /admin/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

The first group repeats the private paths because a named crawler ignores the * group. Naming the search crawlers also means a later Disallow: / under * cannot lock them out by accident.

Google-Extended is left out on purpose, because blocking it would also keep your pages out of grounding in Gemini Apps. If you decide to block it anyway, add User-agent: Google-Extended to the training group. ChatGPT-User and Perplexity-User are not listed, since their owners say robots.txt may not apply to them.

What robots.txt cannot do

• Keep anything private. RFC 9309 says the rules are not a form of access authorization, and listing a path makes it public. Put private content behind a login.
• Stop user-requested fetches. ChatGPT-User and Perplexity-User may fetch a page that robots.txt disallows, according to OpenAI and Perplexity.
• Remove a page from view everywhere. OpenAI says ChatGPT Atlas may still show the link and title of a disallowed page it learns about elsewhere. Its advice is a noindex tag, which its crawler can only read if it is allowed to fetch the page.
• Control Google's AI features on its own. Google names Googlebot as the robots.txt control for AI features in Search. To limit what is shown, it points to nosnippet, data-nosnippet, max-snippet and noindex. Since 31 August 2026, every site also has a Search generative AI control in Search Console that includes or excludes its pages from AI Overviews, AI Mode and Discover's AI features. It is a Search Console setting, not a robots.txt rule. Google's help page for it says it does not affect AI training and points to Google-Extended for that. Our guide on using Google Search Console for AI search covers the setting.
• Beat a firewall. Google's advice is to make sure crawling is allowed by robots.txt and by any CDN or hosting setup. OpenAI and Perplexity publish their crawlers' IP ranges and ask site owners to allow them.

Check that it works

1. Confirm the file answers with a 200 status:

   curl -s -o /dev/null -w "%{http_code}\n" https://www.example.com/robots.txt
   

2. Read the file exactly as a crawler receives it:

   curl -s https://www.example.com/robots.txt
   

3. Request a page with PerplexityBot's published user-agent string:

   curl -s -o /dev/null -w "%{http_code}\n" \
     -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
     https://www.example.com/pricing
   

   A 403 or a challenge page means something in front of your site blocks that name. The test covers the name only: a rule keyed to IP addresses can treat the real crawler differently.

4. Search your server logs for the crawler names:

   grep -E "OAI-SearchBot|Claude-SearchBot|Claude-User|PerplexityBot" access.log | tail -20
   

5. Wait a day after any change, then run the checks again.

A crawler that is allowed in still has to find your content in the HTML it receives. JavaScript and AI crawlers covers that half, and our test of nine pricing pages read without JavaScript shows what goes missing.

What to do first

• List every AI crawler name in your current robots.txt and compare it with the table above.
• Make sure OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot can reach the pages you want cited, in robots.txt and in your firewall.
• Decide on GPTBot and ClaudeBot as one question about training, and on Google-Extended knowing it also covers Gemini grounding.
• Open Settings, then Search generative AI, in Search Console and confirm the setting is the one you want.
• Run the curl checks again after every change.