Robots.txt Tester: Test any URL against any crawler
Fetch a live robots.txt or paste your own. Test a URL for Googlebot, Bingbot or any AI crawler. See the rule that decides it, and test edits before you publish.
Sample file. Fetch a live site or paste your own.
Sitemaps declared: https://example.com/sitemap.xml
Testing /admin/settings
Blocked
Googlebot may not crawl /admin/settings
- Group used
- User-agent: *
- Deciding rule
Every crawler for /admin/settings
- GooglebotBlocked
- BingbotBlocked
- ApplebotBlocked
- DuckDuckBotBlocked
- OAI-SearchBotBlocked
- Claude-SearchBotBlocked
- PerplexityBotBlocked
- DuckAssistBotBlocked
- ChatGPT-UserBlocked
- Claude-UserBlocked
- Perplexity-UserBlocked
- GPTBotBlocked
- ClaudeBotBlocked
- Google-ExtendedBlocked
- Applebot-ExtendedBlocked
- Meta-ExternalAgentBlocked
- CCBotBlocked
- BytespiderBlocked
Using the Robots.txt Tester
- 1
Load a robots.txt
Fetch it from any site, or paste a draft. It stays editable.
- 2
Pick a URL and crawler
Choose Googlebot, an AI crawler or a custom user agent.
- 3
See why
Get the verdict, the group used and the deciding line, plus syntax problems.
How crawlers read robots.txt
Four rules decide almost every result.
The most specific group applies
A crawler uses the group that names it. Googlebot-Image falls back to a Googlebot group, then to User-agent: *. Groups for the same bot merge.
The longest match wins
If several rules match a URL, the longest path wins. On a tie, Allow beats Disallow.
Wildcards are limited
* matches any characters. $ marks the end of the URL. Everything else is a plain prefix, and paths are case-sensitive.
Blocked is not deindexed
Disallow stops crawling, not indexing. A blocked URL can still appear in search if other pages link to it. Use noindex for that, and leave the page crawlable.
Check it on every page. Fix it at the source.
Test one URL here. Catch issues across whole sites with the platform: one plan for in-house teams and agencies.
Check while you browse
The free Chrome extension runs robots.txt, sitemap and llms.txt checks on the page you have open.
Get the extensionEdit robots.txt in WordPress
The WordPress plugin has a robots.txt editor with per-bot AI crawler control. Change rules without FTP.
See the pluginCheck it on every crawl
The Desktop SEO Spider runs 13 robots.txt checks in every crawl. Connect Claude over MCP and it can list the URLs robots.txt blocks.
Explore the Spider