Robots.txt
A text file that tells search engine crawlers which pages they can and cannot access.
The Definition
Robots.txt is a plain text file placed at the root of your website (e.g., example.com/robots.txt) that provides instructions to web crawlers about which URLs they are allowed or disallowed from crawling. It uses the Robots Exclusion Protocol and can specify rules per user agent, including crawl-delay directives.
Why It Matters
A misconfigured robots.txt can block search engines from crawling critical pages, or waste crawl budget by allowing access to low-value URLs. It is the first file search engines check when visiting your site, making it a foundational element of technical SEO.
Best Practices
Test robots.txt changes with Google Search Console robots.txt tester before deploying
Include a Sitemap directive pointing to your XML sitemap for efficient crawl discovery
Be specific with Disallow rules — overly broad patterns like Disallow: / blocks your entire site
Do not use robots.txt to hide sensitive content — use proper authentication instead, as URLs can still appear in search results
Keep your robots.txt file at the root domain level and ensure it loads within 5 seconds
Review crawl stats in GSC to identify if your robots.txt is blocking important crawl paths
Mistakes to Avoid
- 1
Blocking CSS and JavaScript files that Googlebot needs to render your pages properly
- 2
Using robots.txt to prevent indexing — it only blocks crawling, not indexing (use noindex for that)
- 3
Having syntax errors like missing colons or inconsistent spacing that invalidate rules
- 4
Forgetting to update robots.txt after site restructuring, blocking new URL patterns
Audit Checks
How Digispot AI identifies and fixes related issues
Your robots.txt is specifically blocking Googlebot.
Impact: If Googlebot is blocked, your site won't appear in Google search at all.
Remove the rule disallowing Googlebot. • Ensure Googlebot can reach your public pages. • Re-test in Search Console.
Your robots.txt is blocking the entire site from being crawled.
Impact: A fully blocked site can't be crawled or indexed — it will disappear from search.
Find and remove the "Disallow: /" rule blocking everything. • Allow crawlers to access your public content. • Re-test in Search Console immediately.
Your robots.txt returns different rules depending on whether a mobile, desktop, or tablet crawler requests it.
Impact: Inconsistent rules across bots can block mobile-first indexing while allowing desktop (or vice versa).
Check your server/CDN for user-agent-based rules (e.g. Cloudflare bot rules). • Serve one identical robots.txt to all crawlers. • Re-test from multiple user-agents.
robots.txt returned a 5xx server error (or persistently timed out).
Impact: Google halts crawling of the entire site while robots.txt is unreachable with a server error.
Ensure robots.txt returns 200 (with valid rules) or 404 — never a 5xx.
Your robots.txt is blocking a critical crawler.
Impact: Blocking an important bot can stop your content being crawled and indexed where it matters.
Identify which critical bot is being blocked. • Remove or narrow the disallow rule. • Re-test that the bot can crawl.
The sitemap line in robots.txt points to an invalid URL.
Impact: A bad sitemap reference means crawlers can't use it to find your pages.
Correct the Sitemap URL to a valid, absolute address. • Confirm the sitemap loads. • Re-publish robots.txt.