Do the Sites AI Cites Block AI Crawlers? We Checked 100 of Them
We read robots.txt on the 100 sites AI engines cite most. 92 of 98 let every AI search crawler in; six block some and are cited anyway.
Published November 21, 2026 · AEO Spotlight Editorial
The short answer
- 92 of the 98 sites we could read (94%) let all five AI search crawlers in: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Googlebot and Bingbot.
- Training crawlers are blocked more often than search crawlers. 14 of 98 block at least one training crawler. Bytespider is the most blocked (13), then Amazonbot (10), CCBot and Applebot-Extended (9 each), and GPTBot (7).
- Six heavily cited sites block one or more AI search crawlers, and AI engines cite them anyway: Reddit, Forbes, PCMag (and its UK site), ZDNet and LinkedIn.
- 47 of the 100 sites serve an llms.txt file. Many of them are software companies, so read this as "common in software", not "common on the web".
What we checked
We took the 100 host names AI engines cited most often in AEO Spotlight's own monitoring data over the 90 days to November 21st, 2026, across eight engines (ChatGPT, Gemini, Google AI Overviews, Perplexity, DeepSeek, Grok, Claude and Microsoft Copilot). We left out our own site. On November 21st we fetched each site's robots.txt and llms.txt once, one site per second, and resolved every crawler with the robots.txt standard (RFC 9309): a crawler's own group beats User-agent: *, and the longest matching rule wins. It's the same code as our free AI Crawler Checker.
98 of the 100 sites returned a readable answer (95 robots.txt files, 1 missing file, which means "allow everything", and 2 servers that refused our request). One returned a web page instead of robots.txt and one didn't respond.
Search crawlers: almost everyone lets them in
| Crawler | What it does | Sites blocking it (of 98) |
|---|---|---|
| OAI-SearchBot | ChatGPT search | 4 |
| PerplexityBot | Perplexity search | 6 |
| Claude-SearchBot | Claude search | 4 |
| Googlebot | Google Search and AI Overviews | 1 |
| Bingbot | Bing, which Copilot draws on | 1 |
Training crawlers: blocked more, but still by a minority
| Crawler | Sites blocking it (of 98) |
|---|---|
| Bytespider (ByteDance) | 13 |
| Amazonbot | 10 |
| CCBot (Common Crawl) | 9 |
| Applebot-Extended (opt-out switch) | 9 |
| anthropic-ai (legacy) | 8 |
| Meta-ExternalAgent | 8 |
| GPTBot (OpenAI) | 7 |
| ClaudeBot (Anthropic) | 6 |
| Google-Extended (opt-out switch) | 3 |
The six sites that block AI search crawlers, and are cited anyway
| Site | Search crawlers blocked in robots.txt | Citations in our data (90 days) | Engines citing it most |
|---|---|---|---|
| reddit.com | All five (User-agent: * / Disallow: /) |
3,242 | ChatGPT 2,480, Gemini 422, AI Overviews 275 |
| forbes.com | PerplexityBot | 489 | Perplexity 289, Copilot 58, Gemini 52 |
| pcmag.com | OAI-SearchBot, PerplexityBot, Claude-SearchBot | 149 | Perplexity 73, Gemini 35 |
| uk.pcmag.com | OAI-SearchBot, PerplexityBot, Claude-SearchBot | 57 | Perplexity 45, Grok 12 |
| zdnet.com | OAI-SearchBot, PerplexityBot, Claude-SearchBot | 68 | Perplexity 41, Grok 27 |
| linkedin.com | PerplexityBot | 130 | AI Overviews 76, Perplexity 39 |
What this shows, and what it doesn't:
- robots.txt controls crawling, not citing. An engine can name a page it learned about another way: a content licensing deal (Reddit has public data agreements with Google and OpenAI), a search partner's index, or a person pasting a link. Our data can't tell which route each citation took.
- The public file isn't always the whole policy. Reddit refused our request when we sent a Googlebot user agent from our own server, which suggests it checks who is really asking. The real Googlebot may see different rules. We report the file the public sees.
- Where the block and the engine line up, citations are rare. ChatGPT cited PCMag and ZDNet zero times in our data; both block OAI-SearchBot. It cited Forbes, which allows OAI-SearchBot, only 5 times, so being allowed in doesn't mean being cited either.
What this means for your site
- Check that the five search crawlers are allowed. Almost every site AI cites allows them, and a search-crawler block is the robots.txt setting most likely to keep you out of answers. Run the free AI Crawler Checker.
- Blocking training crawlers is a business choice. Blocking GPTBot or ClaudeBot doesn't remove you from ChatGPT or Claude search, which use different crawlers.
- Access is the first step, not the result. Whether an engine names you depends on what it finds. Our free Brand Mention Check shows what ChatGPT and Perplexity say about your brand today.
Limits
- A sample of our data, not the whole web. The 100 sites come from the questions monitored in our own accounts, a mix of internal, pilot and demo brands (software, marketing tools, ecommerce, CRM, project management and running gear), so the list leans toward those categories.
- One snapshot. Each robots.txt was read once on November 21st, 2026, from one server with our own user agent. Files change, and some sites serve different rules to verified crawlers.
- We counted host names. forbes.com and blog.hubspot.com are separate rows, because each host has its own robots.txt.
- Citations aren't crawl logs. We record which pages engines name in their answers, not how they got them.
Frequently Asked Questions
Do the sites AI cites block AI crawlers?
Mostly not. Of the 98 most-cited sites we could read in November 2026, 92 let all five AI search crawlers in. Training crawlers were blocked more often.
Can a site that blocks AI crawlers still be cited?
Yes. Reddit, Forbes, PCMag, ZDNet and LinkedIn block one or more AI search crawlers and were still cited in our data. A citation can come from a licensing deal, a search partner's index or a pasted link, and our data can't tell which.