Free technical SEO tool

Robots.txt Tester: Test any URL against any crawler

Fetch a live robots.txt or paste your own. Test a URL for Googlebot, Bingbot or any AI crawler. See the rule that decides it, and test edits before you publish.

Try:
No problems

Sample file. Fetch a live site or paste your own.

Sitemaps declared: https://example.com/sitemap.xml

Testing /admin/settings

Blocked

Googlebot may not crawl /admin/settings

Group used
User-agent: *
Deciding rule

Every crawler for /admin/settings

  • GooglebotBlocked
  • BingbotBlocked
  • ApplebotBlocked
  • DuckDuckBotBlocked
  • OAI-SearchBotBlocked
  • Claude-SearchBotBlocked
  • PerplexityBotBlocked
  • DuckAssistBotBlocked
  • ChatGPT-UserBlocked
  • Claude-UserBlocked
  • Perplexity-UserBlocked
  • GPTBotBlocked
  • ClaudeBotBlocked
  • Google-ExtendedBlocked
  • Applebot-ExtendedBlocked
  • Meta-ExternalAgentBlocked
  • CCBotBlocked
  • BytespiderBlocked
How it works

Using the Robots.txt Tester

  1. 1

    Load a robots.txt

    Fetch it from any site, or paste a draft. It stays editable.

  2. 2

    Pick a URL and crawler

    Choose Googlebot, an AI crawler or a custom user agent.

  3. 3

    See why

    Get the verdict, the group used and the deciding line, plus syntax problems.

Know the rules

How crawlers read robots.txt

Four rules decide almost every result.

The most specific group applies

A crawler uses the group that names it. Googlebot-Image falls back to a Googlebot group, then to User-agent: *. Groups for the same bot merge.

The longest match wins

If several rules match a URL, the longest path wins. On a tie, Allow beats Disallow.

Wildcards are limited

* matches any characters. $ marks the end of the URL. Everything else is a plain prefix, and paths are case-sensitive.

Blocked is not deindexed

Disallow stops crawling, not indexing. A blocked URL can still appear in search if other pages link to it. Use noindex for that, and leave the page crawlable.

Beyond one check

Check it on every page. Fix it at the source.

Test one URL here. Catch issues across whole sites with the platform: one plan for in-house teams and agencies.

FAQ

Questions, answered

More free AI-search tools