AI Crawler Checker: See which AI bots can read your site
Enter a site. We read its robots.txt and check 18 crawlers, from GPTBot and ClaudeBot to Googlebot. Every verdict shows the rule behind it.
Using the AI Crawler Checker
- 1
Enter a website
Any public domain. We fetch its live /robots.txt.
- 2
Read the verdicts
AI search, user fetches, training and search engines, each allowed or blocked.
- 3
Trace the rule
Click a verdict to jump to the exact line. Test any path, not just the homepage.
Not every AI crawler does the same job
Blocking the wrong bot can take you out of AI answers. Know the four kinds.
AI search crawlers
OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so assistants can cite and link you. Block them and those answers skip you.
Training crawlers
GPTBot, ClaudeBot, CCBot and others collect training data. Blocking them is a policy choice. It does not remove you from AI search.
Control tokens
Google-Extended and Applebot-Extended are not crawlers. They tell Google whether content may train and ground Gemini, and Apple whether it may train its models. Search is unaffected.
robots.txt is a request
Reputable crawlers follow it. A CDN or firewall rule can still block bots by user agent, and that never shows in robots.txt.
Check it on every page. Fix it at the source.
One plan covers all four products. Agencies use it to check AI access across every client site.
Check any page you visit
The free Chrome extension shows AI crawler access for 25 bots, plus robots.txt, sitemap and llms.txt checks, on the page you have open.
Get the extensionFix it on WordPress
The WordPress plugin includes a robots.txt editor with per-bot control for 23 AI crawlers. No file editing.
See the pluginCompare with competitors
The Desktop SEO Spider (Mac and Windows) checks robots.txt and llms.txt in every crawl, and profiles each rival's AI-crawler stance from their robots.txt.
Explore the Spider