Robots.txt Issues

The robots.txt file is the first thing search engine crawlers read when they visit your domain — and mistakes here can have site-wide consequences that are notoriously difficult to diagnose. A single overly broad Disallow rule can block Google from crawling entire sections of your site, effectively de-indexing hundreds of pages overnight. Conversely, a missing or permissive robots.txt can waste your crawl budget by letting bots crawl admin panels, search results pages, and other low-value URLs. The robots.txt specification (RFC 9309, standardized in 2022) is deceptively simple, but real-world implementations frequently contain syntax errors, conflicting directives for different user agents, or rules that inadvertently block CSS and JavaScript files critical for rendering. Google has also clarified that robots.txt blocking is advisory for crawling but does not prevent indexing — if other pages link to a blocked URL, Google may still index it without crawling its content, creating 'indexed but not crawled' anomalies in Search Console. This section documents every robots.txt issue Digispot AI detects, from missing files to complex directive conflicts.