Which AI Crawlers Should You Allow in robots.txt? (2026)
Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot for AI search; blocking GPTBot or ClaudeBot only affects training.
Published November 21, 2026 · AEO Spotlight Editorial
If you want AI engines to find and link to your pages, allow their search crawlers: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Googlebot (Google Search and AI Overviews) and Bingbot (Bing, which Microsoft Copilot draws on). Blocking training crawlers such as GPTBot or ClaudeBot is a separate business choice; it does not remove you from AI search.
What are the three kinds of AI crawler?
AI companies run different crawlers for different jobs, and robots.txt treats each one separately. Most confusion about "blocking AI" comes from treating them as one thing.
| Kind | What it does | Examples |
|---|---|---|
| Search and citations | Builds the index an AI engine searches and links to in answers | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot |
| User-triggered fetch | Opens a page when a person asks the assistant to | ChatGPT-User, Perplexity-User, Claude-User |
| Training only | Collects text to train future models | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot |
There is also a fourth thing that looks like a crawler but isn't: opt-out tokens. Google-Extended and Applebot-Extended don't fetch anything. They tell Google and Apple whether content their normal crawlers already collected may be used for AI training.
Which crawlers decide whether AI search can see you?
The search-and-citation crawlers. If robots.txt blocks OAI-SearchBot, OpenAI documents that your pages won't be shown in ChatGPT search answers (they can still appear as plain navigational links). The same logic applies to PerplexityBot for Perplexity and Claude-SearchBot for Claude. Googlebot covers Google Search and the AI Overviews built on it, and Bingbot covers the Bing index that Copilot draws on.
Blocking one of these is the robots.txt setting most likely to keep you out of AI answers. It's also easy to do by accident, for example with a User-agent: * / Disallow: / left over from a staging site.
To check by hand, open yoursite.com/robots.txt, look for a group naming each of the five search crawlers, then read the User-agent: * group, which applies to any crawler without its own group.
Should you block GPTBot and ClaudeBot?
It depends on whether you're comfortable with your content being used to train models; it doesn't affect AI search. GPTBot and ClaudeBot collect training data. OpenAI and Anthropic run separate crawlers (OAI-SearchBot and Claude-SearchBot) for search, and each one follows its own robots.txt group.
So a site can allow ChatGPT search and block OpenAI training at the same time:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
What about Google-Extended?
Google-Extended doesn't affect Google Search or AI Overviews. Google documents it as a switch for whether your content may be used to train Gemini and Vertex AI models. Googlebot still crawls and indexes the site either way. Applebot-Extended works the same way for Apple Intelligence training.
So blocking Google-Extended is a training decision, like blocking GPTBot. If you want to appear in AI Overviews, the setting that matters is Googlebot: a Disallow for Googlebot stops Google crawling the pages both Search and AI Overviews draw on.
How does robots.txt decide which rule applies?
It follows the robots.txt standard (RFC 9309), and three rules matter most:
- A crawler's own group beats
User-agent: *. If your file has aUser-agent: PerplexityBotgroup, PerplexityBot ignores the*group entirely, even if the*group is stricter or looser. - The longest matching path wins.
Disallow: /blog/andAllow: /blog/guides/together let crawlers into/blog/guides/but not the rest of/blog/. - A missing robots.txt means everything is allowed. If your server answers with a server error (5xx) instead, crawlers that follow the standard stop until it works again.
The first rule causes the most surprises. A site that adds User-agent: GPTBot / Disallow: /private/ has just given GPTBot its own group, so the site-wide rules under * no longer apply to it.
A robots.txt that allows AI search but not training
This is a common middle ground. Adjust the training list to suit your own policy:
# AI search and citations: allowed
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
# AI training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Several User-agent lines in a row share the group below them, so each block above applies to every crawler it names.
What do the sites AI cites most actually do?
Most let AI search crawlers in. When we checked the robots.txt of the 100 sites AI engines cited most in our monitoring data (November 2026), 92 of the 98 we could read allowed all five search crawlers. Training crawlers were blocked more often: 7 blocked GPTBot and 13 blocked Bytespider. The full study has the method and its limits.
Does allowing crawlers mean AI will mention you?
No. Access is the first step: it lets an engine fetch your pages. Whether it names you depends on what it finds and on what other sources say. Some heavily cited sites block AI search crawlers and are cited anyway; an engine can learn about a page in other ways, such as a licensing deal or a search partner's index.
In our study (November 2026), ChatGPT cited PCMag and ZDNet zero times in 90 days; both block OAI-SearchBot. It cited Forbes, which allows OAI-SearchBot, only 5 times. AI answers also vary from run to run, so watch the same questions over weeks rather than trusting one check.
Next step
Run the free AI Crawler Checker on your site. It reads your robots.txt the way these crawlers do, sorts them into the groups above, and shows the line to change if a search crawler is blocked.
Frequently Asked Questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, which follows its own robots.txt group, so you can block one and allow the other.
Do I need a separate rule for every AI crawler?
No. If robots.txt has no group for a crawler, the User-agent: * group applies to it. You only need a named group when you want a crawler treated differently from everyone else.
Do ChatGPT-User and Perplexity-User follow robots.txt?
These fetch a page when a person asks the assistant to. Some vendors say their user-triggered fetchers may not always follow robots.txt, so don't rely on robots.txt alone to keep a page private.
How often should I check my robots.txt for AI crawlers?
Check after any site migration, CMS or CDN change, or bot-protection change. Those are the moments a staging rule or a firewall setting most often starts blocking crawlers by accident.