Robots.txt for AI Crawlers: Should You Block GPTBot, ClaudeBot and PerplexityBot?
Which AI crawlers power search answers and which collect training data, what each vendor says happens if you block them, and a copy-ready robots.txt for ecommerce brands that want AI visibility.
Most AI companies run separate crawlers for search and for model training, and you can control them independently in robots.txt. If you want your products to appear in AI answers, allow the search crawlers, such as OAI-SearchBot (ChatGPT search), PerplexityBot and Claude-SearchBot, plus Googlebot for Google's AI Overviews and AI Mode. Training crawlers like GPTBot, ClaudeBot and the Google-Extended token are a separate decision.
- Search crawlers and training crawlers are separate, and each vendor lets you control them independently.
- Blocking OAI-SearchBot keeps your site out of ChatGPT search answers, according to OpenAI.
- PerplexityBot is for search results and, per Perplexity, not for training foundation models.
- Google-Extended doesn't affect inclusion or ranking in Google Search; AI Overviews use normal Googlebot crawling.
- Check your CDN or firewall too: a WAF rule can block a bot even when robots.txt allows it.
On this page
A lot of brands blocked "AI bots" in 2023 and 2024, often with a single copy-pasted block list. Some of those same brands now wonder why ChatGPT or Perplexity never cites their product pages.
The problem is that "AI crawler" covers very different things. Some crawlers collect content to train models. Others index pages so an assistant can cite and link to you in answers. A third kind fetches a page because a user asked. Each vendor documents these separately, and you can make a different choice for each.
Which AI crawlers matter for ecommerce brands?
Here's what each major vendor says about its crawlers, in its own documentation.
| Vendor | User agent | Purpose (per vendor docs) |
|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces websites in ChatGPT search. Sites opted out won't be shown in ChatGPT search answers. |
| OpenAI | GPTBot | Crawls content that may be used to train foundation models. |
| OpenAI | ChatGPT-User | Visits pages for user actions; robots.txt rules may not apply. Not used to decide what appears in search. |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search results; not used to crawl for AI foundation models. |
| Perplexity | Perplexity-User | Fetches pages for user questions; generally ignores robots.txt. |
| Anthropic | Claude-SearchBot | Indexes content to improve search results; blocking may reduce visibility in user search results. |
| Anthropic | Claude-User | Fetches pages when users ask Claude questions; blocking may reduce visibility for user-directed search. |
| Anthropic | ClaudeBot | Collects content that could contribute to model training. |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode. | |
| Storebot-Google | Crawls for Google's shopping surfaces. | |
| Google-Extended | A control token, not a separate crawler: governs use for training and grounding of Gemini models. Doesn't affect Search inclusion or ranking. |
Sources: OpenAI, Perplexity, Anthropic, Google.
What happens if you block each one?
The vendors are fairly explicit:
- OAI-SearchBot: OpenAI says sites opted out won't be shown in ChatGPT search answers, though they can still appear as navigational links. It recommends allowing OAI-SearchBot and its published IP ranges.
- GPTBot: Disallowing it signals that your content shouldn't be used to train OpenAI's foundation models. OpenAI says the settings are independent, so you can allow OAI-SearchBot while disallowing GPTBot.
- PerplexityBot: Perplexity recommends allowing it to make sure your site appears in search results.
- Claude-SearchBot and Claude-User: Anthropic says disabling these may reduce your visibility in user search results.
- Google-Extended: Google says it does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode use normal Search crawling, and Google's AI features guide lists allowing crawling in robots.txt and in your CDN or hosting infrastructure as a best practice.
What should an ecommerce brand's robots.txt look like?
Here's a starting point for a brand that wants maximum AI search visibility and has decided to allow training as well. Adjust the training lines if your policy differs.
# Search engines and AI search crawlers: allow
User-agent: Googlebot
Allow: /
User-agent: Storebot-Google
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# User-triggered fetchers
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
# Training (a separate policy decision)
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
# Everyone else
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Sitemap: https://www.example.com/sitemap.xml
If you want to opt out of training but stay visible in AI search, change the three training groups to Disallow: / and leave the search crawlers allowed.
Is robots.txt enough?
Often not. Three other places can silently block AI crawlers:
- Your CDN or WAF. Bot-management and firewall rules can block a crawler before it ever reads robots.txt. Perplexity's documentation includes WAF allowlisting guidance that combines user-agent and IP checks. OpenAI publishes IP ranges for its crawlers too.
- Rendering. If your product details only appear after heavy client-side JavaScript, some crawlers may see an empty page. Put key product facts in the HTML.
- Meta robots and snippet controls. Google's AI features respect
noindex,nosnippet,data-nosnippetandmax-snippet. A straynosnippeton product templates can keep pages out of AI Overviews.
How do you check whether AI crawlers can reach your site?
- Open yoursite.com/robots.txt and look for named groups or blanket
Disallow: /rules affecting AI crawlers. - Check CDN and WAF bot settings for rules that block AI user agents or their published IP ranges.
- Search server logs for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot hits on product pages.
- View page source on a product page: are price, specs and description in the HTML?
- Check product templates for unintended noindex or nosnippet directives.
- After changes, allow about 24 hours (per OpenAI and Perplexity), then re-test in each assistant.
Do I need llms.txt too?
llms.txt is a proposed standard for a Markdown file that summarizes your site for language models. It's cheap to add, and some tools read it. But Google says you don't need new machine-readable files to appear in its AI features. Treat it as optional, and never as a substitute for crawlable pages. We cover this and other misconceptions in 7 AEO Myths.
What should you do next?
Fixing crawler access is usually the fastest technical AEO win, because a blocked page can't be cited at all. Once access is sorted, move on to product data: our AI-ready listing checklist covers it, and How ChatGPT chooses products explains what happens next.
“Product schema, review markup, canonical tags, index status, llms.txt, and machine-readable specifications, checked on every tracked product page.”
Frequently asked questions
If I block GPTBot, will I disappear from ChatGPT?
Not from ChatGPT search, according to OpenAI. GPTBot controls whether content may be used to train OpenAI's foundation models. OAI-SearchBot is the crawler that surfaces sites in ChatGPT search, and the two settings are independent.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not impact a site's inclusion in Google Search, and AI Overviews and AI Mode draw on pages indexed by normal Search crawling. Google-Extended controls use for training and grounding of Gemini models.
How long do robots.txt changes take to apply?
OpenAI says about 24 hours for ChatGPT search, and Perplexity says up to 24 hours. Other crawlers re-read robots.txt on their own schedules.
Do user-triggered fetchers follow robots.txt?
It depends on the vendor. OpenAI says robots.txt rules may not apply to ChatGPT-User because actions are user-initiated, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says Claude-User respects robots.txt and that blocking it may reduce visibility for user-directed search.
Find out which of your products AI recommends.
Score a product across ChatGPT, Gemini, Perplexity, Copilot and Google AI Overviews with a free SearchPanel audit. Or book a meeting and we'll walk through a live AI Visibility Score with you.
Sources
- OpenAI: Overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Claude Help Center: Does Anthropic crawl data from the web?
- Google Search Central: Google's common crawlers
- Google Search Central: AI features and your website
XENA results quoted in this article are as published by XENA Intelligence on the linked pages. Third-party facts are from the linked official documentation, checked Oct 26, 2026.
