For publishers

Which AI crawlers do you block, and which AI engines cite you?

Enter your site. We read your public robots.txt using the robots.txt standard (RFC 9309), then count how often eight AI engines cited your site in our monitoring data over the last 90 days. Free, no account needed.

Each host is counted on its own: uk.example.com is separate from example.com.

Why we built this

On November 21st, 2026 we read the robots.txt of the 100 sites AI engines cited most in our data. 92 of the 98 we could read let every AI search crawler in. Six block at least one and are still cited in our data: Forbes blocks PerplexityBot, and Perplexity cited Forbes 289 times. Our data can't show how an engine reached a page; a licensing deal, a search partner's index or a link a person shared could each explain it. Read the study.

robots.txt decides who may crawl. This lookup puts that next to who names you in answers, for your own site.

How the counts work, and their limits

  • We count each time an engine named a page on your host (and its www. form) as a source in an answer we recorded in the last 90 days.
  • The answers come from the questions monitored in AEO Spotlight's own accounts, across ChatGPT, Perplexity, Gemini, Google AI Overviews, Microsoft Copilot, Grok, Claude and DeepSeek. It's a sample, not the whole web, and it leans toward software, marketing and ecommerce topics.
  • A citation shows what an engine named, not how it got the page. We don't see crawl logs.
  • We read the robots.txt the public sees, once, when you look up the site. Some sites show verified crawlers different rules, and a firewall can block crawlers that robots.txt allows.
  • We never show which questions or which accounts produced a citation.

Want the raw counts for your site, or a look at specific topics? Email support@aeospotlight.com.