Meta Robots Tag
An HTML meta tag that controls how search engines crawl and index a page.
The Definition
The meta robots tag is an HTML element placed in the HTML head section that provides directives to search engine crawlers. Common values include 'index/noindex' (whether to include the page in search results), 'follow/nofollow' (whether to follow links on the page), and 'noarchive' (whether to cache the page).
Why It Matters
Incorrect meta robots directives can accidentally block important pages from being indexed or prevent search engines from following valuable internal links. A single misplaced 'noindex' tag can remove a high-traffic page from search results entirely.
Best Practices
Use noindex only on pages you genuinely want excluded from search results — tag pages, internal search results, admin pages
Check for X-Robots-Tag HTTP headers that may conflict with or override your HTML meta robots directives
Verify robots directives align with your canonical strategy and sitemap — inconsistencies confuse search engines
Use nofollow sparingly on internal links — it wastes PageRank rather than directing it
Test robots directives in staging environments before deploying to production to avoid accidentally deindexing pages
Regularly audit robots directives after CMS or theme updates, which frequently add unwanted noindex tags
Mistakes to Avoid
- 1
Leaving a staging environment noindex directive in place when migrating to production
- 2
Using noindex on paginated pages instead of proper canonical and pagination markup
- 3
Setting noindex, follow and expecting links to be followed — Google may eventually stop crawling noindex pages entirely
- 4
Conflicting directives between meta tags and HTTP headers causing unpredictable indexing behavior
Audit Checks
How Digispot AI identifies and fixes related issues
A meta robots tag is sitting in the page body instead of the <head>.
Impact: Crawlers only read robots directives in the <head> — one in the body is silently ignored, so your index/follow rule never applies.
Move the <meta name="robots"> tag inside the <head>. • Make sure no script injects it into the body after load. • Re-crawl to confirm the directive is now honored.
The meta robots tag uses the "none" directive, equivalent to noindex,nofollow.
Impact: A "none" directive removes the page from search results and stops link-following.
Remove the "none" directive unless the page is intentionally de-indexed.
The meta robots tag has directives that contradict each other (e.g. index and noindex).
Impact: Conflicting rules make crawl/index behavior unpredictable — Google may pick the restrictive one.
Decide the intended rule for the page. • Keep one consistent set (e.g. index, follow). • Remove the contradicting directive.
The robots tag has an "unavailable_after" date that has already passed.
Impact: An expired date can stop the page being indexed as expected.
Update or remove the unavailable_after date. • Set it only if you truly want the page to expire. • Re-crawl to confirm.
The robots tag prevents search snippets from showing.
Impact: No snippet means your result has no description, which can lower clicks.
Confirm you really want to suppress snippets. • If not, allow them (e.g. set a max-snippet length). • Re-crawl to confirm.
The robots tag has conflicting snippet directives (max-snippet vs nosnippet).
Impact: Conflicting directives make snippet behavior unpredictable.
Decide whether you want snippets or not. • Keep one consistent snippet directive. • Re-crawl to confirm.