Robots.txt Generator: With controls for every AI crawler
Pick a policy, fine-tune each crawler, keep private paths out and add your sitemap. The file is checked as you build it.
2. Fine-tune each crawler
AI search & answers
These index pages so AI assistants can cite and link you. Blocking them removes you from those answers.
AI user-triggered fetches
A person asked an assistant to open your page. Some of these may not follow robots.txt.
AI model training
These collect content to train models. Blocking them is a policy choice. It does not affect search or AI citations.
Search engines
Block these and you drop out of search results, including Google AI Overviews and Bing Copilot answers.
3. Keep private paths out (all crawlers)
Use the full URL. Crawlers find your sitemap faster.
Your robots.txt
# Generated with Digispot AI (digispot.ai) User-agent: * Disallow: # Crawlers you chose to block User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot User-agent: Bytespider Disallow: /
Checked with the same parser as our robots.txt tester: no problems. Upload it to your site root as /robots.txt.
Using the Robots.txt Generator
- 1
Choose a policy
Allow everything, allow AI search but block training, or block all AI crawlers.
- 2
Fine-tune
Switch any crawler, add private paths and your sitemap URL.
- 3
Copy or download
Upload the file to your site root as /robots.txt. Test it with our tester.
A good robots.txt is short
Block what crawlers should never fetch. Leave the rest open.
Never block CSS and JavaScript
Google renders pages. Blocking assets can make pages look broken to it.
Block low-value URLs, not pages you want gone
Cart, checkout, internal search and admin paths waste crawl budget. Pages you want out of search need noindex instead.
Decide AI search and training separately
AI search crawlers bring citations and links. Training crawlers do not. Allow the first if you want AI citations, then choose on the second.
List your sitemap
A Sitemap line helps every crawler find your URLs. Use the full https:// address.
Check it on every page. Fix it at the source.
Publish it, then keep it right. One plan covers all four products, for in-house teams and agencies.
Manage it in WordPress
The WordPress plugin has a robots.txt editor with per-bot control for 23 AI crawlers, plus llms.txt and sitemaps.
See the pluginVerify on any page
The free Chrome extension shows AI crawler access for 25 bots on the page you have open.
Get the extensionRe-crawl after publishing
The Desktop SEO Spider checks robots.txt, sitemaps and llms.txt in every crawl and flags robots.txt problems.
Explore the Spider