Free AI search tool

AI Crawler Checker: See which AI bots can read your site

Enter a site. We read its robots.txt and check 18 crawlers, from GPTBot and ClaudeBot to Googlebot. Every verdict shows the rule behind it.

Try:

How it works

Using the AI Crawler Checker

  1. 1

    Enter a website

    Any public domain. We fetch its live /robots.txt.

  2. 2

    Read the verdicts

    AI search, user fetches, training and search engines, each allowed or blocked.

  3. 3

    Trace the rule

    Click a verdict to jump to the exact line. Test any path, not just the homepage.

Know the rules

Not every AI crawler does the same job

Blocking the wrong bot can take you out of AI answers. Know the four kinds.

AI search crawlers

OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so assistants can cite and link you. Block them and those answers skip you.

Training crawlers

GPTBot, ClaudeBot, CCBot and others collect training data. Blocking them is a policy choice. It does not remove you from AI search.

Control tokens

Google-Extended and Applebot-Extended are not crawlers. They tell Google whether content may train and ground Gemini, and Apple whether it may train its models. Search is unaffected.

robots.txt is a request

Reputable crawlers follow it. A CDN or firewall rule can still block bots by user agent, and that never shows in robots.txt.

Beyond one check

Check it on every page. Fix it at the source.

One plan covers all four products. Agencies use it to check AI access across every client site.

FAQ

Questions, answered

More free AI-search tools