AI Crawler Checker

Can ChatGPT, Perplexity, Claude, Google and Bing reach your site? Enter your address. We read your robots.txt the way the crawlers do and show which AI crawlers it lets in, which it blocks, and the line that does it. Free, no account needed.

Enter a page address to also check that page's path.

How it works

  1. We fetch your robots.txt and llms.txt, as a plain request with no JavaScript.
  2. For each crawler we find the group that applies to it, following the robots.txt standard (RFC 9309): a crawler's own group beats User-agent: *, and the longest matching rule wins.
  3. We sort the crawlers by what they do, because blocking a training crawler and blocking a search crawler have very different effects.

We read the robots.txt the public sees. A few large sites show verified crawlers (for example, the real Googlebot) different rules. A firewall or bot protection service can still block crawlers that robots.txt allows; the site scan fetches your pages the way they do.

Questions

Which crawlers matter for AI answers?

For AI search and citations: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Googlebot (Google Search and AI Overviews) and Bingbot (Bing, which Microsoft Copilot draws on). GPTBot and ClaudeBot only collect training data.

Should I block GPTBot?

That is a business choice. GPTBot collects training data for OpenAI models. Blocking it does not remove you from ChatGPT search, which uses OAI-SearchBot.

What if I have no robots.txt?

Under the robots.txt standard (RFC 9309), a missing file means every crawler may fetch every page.

Does Google-Extended affect AI Overviews?

No. Google documents Google-Extended as a training opt-out for Gemini and Vertex AI. Google Search and AI Overviews use Googlebot.

Do I need an llms.txt file?

It is optional. It is a proposed plain-text summary of your site for language models. No major AI engine has confirmed that it reads one.

If every crawler is allowed, will AI cite my site?

Not necessarily. Access is the first step: it lets AI engines fetch your pages. Whether they name you depends on what they find. Our free Brand Mention Check shows what ChatGPT and Perplexity say about your brand today.