Robots.txt Generator — Control AI Crawlers (GPTBot, ClaudeBot & More)
Generate a valid robots.txt in seconds, with explicit rules for AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and more — alongside standard search engine bots. Free, copy-paste ready.
📋 Generated robots.txt
Save the above as /robots.txt at your site root.
Why control AI crawlers?
- 🤖 GPTBot / ClaudeBot / PerplexityBot scrape your content for training or answers — often without linking back
- 🔒 Block directories you do not want indexed (admin, private pages)
- 🗺️ Declare your sitemap so search engines discover pages faster
- ⚖️ Allow selectively: open up for AI search visibility, or block to protect copyright