Free technical SEO tool

Robots.txt Generator: With controls for every AI crawler

Pick a policy, fine-tune each crawler, keep private paths out and add your sitemap. The file is checked as you build it.

AI crawler policy

1. Choose an AI crawler policy

2. Fine-tune each crawler

AI search & answers

These index pages so AI assistants can cite and link you. Blocking them removes you from those answers.

AI user-triggered fetches

A person asked an assistant to open your page. Some of these may not follow robots.txt.

AI model training

These collect content to train models. Blocking them is a policy choice. It does not affect search or AI citations.

Search engines

Block these and you drop out of search results, including Google AI Overviews and Bing Copilot answers.

3. Keep private paths out (all crawlers)

Use the full URL. Crawlers find your sitemap faster.

Your robots.txt

# Generated with Digispot AI (digispot.ai)

User-agent: *
Disallow:

# Crawlers you chose to block
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

Checked with the same parser as our robots.txt tester: no problems. Upload it to your site root as /robots.txt.

How it works

Using the Robots.txt Generator

  1. 1

    Choose a policy

    Allow everything, allow AI search but block training, or block all AI crawlers.

  2. 2

    Fine-tune

    Switch any crawler, add private paths and your sitemap URL.

  3. 3

    Copy or download

    Upload the file to your site root as /robots.txt. Test it with our tester.

Know the rules

A good robots.txt is short

Block what crawlers should never fetch. Leave the rest open.

Never block CSS and JavaScript

Google renders pages. Blocking assets can make pages look broken to it.

Block low-value URLs, not pages you want gone

Cart, checkout, internal search and admin paths waste crawl budget. Pages you want out of search need noindex instead.

Decide AI search and training separately

AI search crawlers bring citations and links. Training crawlers do not. Allow the first if you want AI citations, then choose on the second.

List your sitemap

A Sitemap line helps every crawler find your URLs. Use the full https:// address.

Beyond one check

Check it on every page. Fix it at the source.

Publish it, then keep it right. One plan covers all four products, for in-house teams and agencies.

FAQ

Questions, answered

More free AI-search tools