robots.txt Generator — With AI Crawler Controls (GPTBot, ClaudeBot, PerplexityBot)
Generate a clean robots.txt with one click — set sitemap, disallow paths, and block AI crawlers like GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, and Google-Extended. Copy-paste ready for your site root.
📋 Generated robots.txt
Save the above as /robots.txt at your site root.
Why control AI crawlers?
- 🤖 GPTBot / ClaudeBot / PerplexityBot scrape your content for training or answers — often without linking back
- 🔒 Block directories you do not want indexed (admin, private pages)
- 🗺️ Declare your sitemap so search engines discover pages faster
- ⚖️ Allow selectively: open up for AI search visibility, or block to protect copyright