Research

Do the Sites AI Cites Block AI Crawlers? We Checked 100 of Them

We read robots.txt on the 100 sites AI engines cite most. 92 of 98 let every AI search crawler in; six block some and are cited anyway.

Published November 21, 2026 · AEO Spotlight Editorial

The short answer

What we checked

We took the 100 host names AI engines cited most often in AEO Spotlight's own monitoring data over the 90 days to November 21st, 2026, across eight engines (ChatGPT, Gemini, Google AI Overviews, Perplexity, DeepSeek, Grok, Claude and Microsoft Copilot). We left out our own site. On November 21st we fetched each site's robots.txt and llms.txt once, one site per second, and resolved every crawler with the robots.txt standard (RFC 9309): a crawler's own group beats User-agent: *, and the longest matching rule wins. It's the same code as our free AI Crawler Checker.

98 of the 100 sites returned a readable answer (95 robots.txt files, 1 missing file, which means "allow everything", and 2 servers that refused our request). One returned a web page instead of robots.txt and one didn't respond.

Search crawlers: almost everyone lets them in

Crawler What it does Sites blocking it (of 98)
OAI-SearchBot ChatGPT search 4
PerplexityBot Perplexity search 6
Claude-SearchBot Claude search 4
Googlebot Google Search and AI Overviews 1
Bingbot Bing, which Copilot draws on 1

Training crawlers: blocked more, but still by a minority

Crawler Sites blocking it (of 98)
Bytespider (ByteDance) 13
Amazonbot 10
CCBot (Common Crawl) 9
Applebot-Extended (opt-out switch) 9
anthropic-ai (legacy) 8
Meta-ExternalAgent 8
GPTBot (OpenAI) 7
ClaudeBot (Anthropic) 6
Google-Extended (opt-out switch) 3

The six sites that block AI search crawlers, and are cited anyway

Site Search crawlers blocked in robots.txt Citations in our data (90 days) Engines citing it most
reddit.com All five (User-agent: * / Disallow: /) 3,242 ChatGPT 2,480, Gemini 422, AI Overviews 275
forbes.com PerplexityBot 489 Perplexity 289, Copilot 58, Gemini 52
pcmag.com OAI-SearchBot, PerplexityBot, Claude-SearchBot 149 Perplexity 73, Gemini 35
uk.pcmag.com OAI-SearchBot, PerplexityBot, Claude-SearchBot 57 Perplexity 45, Grok 12
zdnet.com OAI-SearchBot, PerplexityBot, Claude-SearchBot 68 Perplexity 41, Grok 27
linkedin.com PerplexityBot 130 AI Overviews 76, Perplexity 39

What this shows, and what it doesn't:

What this means for your site

  1. Check that the five search crawlers are allowed. Almost every site AI cites allows them, and a search-crawler block is the robots.txt setting most likely to keep you out of answers. Run the free AI Crawler Checker.
  2. Blocking training crawlers is a business choice. Blocking GPTBot or ClaudeBot doesn't remove you from ChatGPT or Claude search, which use different crawlers.
  3. Access is the first step, not the result. Whether an engine names you depends on what it finds. Our free Brand Mention Check shows what ChatGPT and Perplexity say about your brand today.

Limits

Frequently Asked Questions

Do the sites AI cites block AI crawlers?

Mostly not. Of the 98 most-cited sites we could read in November 2026, 92 let all five AI search crawlers in. Training crawlers were blocked more often.

Can a site that blocks AI crawlers still be cited?

Yes. Reddit, Forbes, PCMag, ZDNet and LinkedIn block one or more AI search crawlers and were still cited in our data. A citation can come from a licensing deal, a search partner's index or a pasted link, and our data can't tell which.