Free Tool - No Signup Required

Free Robots.txt Generator

Control which AI crawlers and search bots can access your website. Generate a production-ready robots.txt in seconds.

14
14/14 AI crawlers allowed
Great AI visibility

Default Crawl Rule

Sitemap URL

Adding a sitemap helps crawlers discover your pages

AI Crawlers

GPTBotOpenAI training crawler
Allow
ChatGPT-UserChatGPT live browsing
Allow
ClaudeBotAnthropic Claude crawler
Allow
PerplexityBotPerplexity AI search
Allow
Google-ExtendedGoogle AI training
Allow
AmazonbotAmazon Alexa crawler
Allow
Applebot-ExtendedApple AI features
Allow
CCBotCommon Crawl bot
Allow
anthropic-aiAnthropic AI training
Allow
cohere-aiCohere AI training
Allow
BytespiderByteDance / TikTok crawler
Allow
FacebookBotMeta AI crawler
Allow
GoogleOtherGoogle secondary crawler
Allow
OAI-SearchBotOpenAI search bot
Allow

Custom Rules

Add custom User-agent and Disallow path pairs

robots.txt Preview

User-agent: *
Allow: /

How to use this file

  1. Copy the generated robots.txt above
  2. Create a file named robots.txt in your site root
  3. Paste the contents and save
  4. Verify at yourdomain.com/robots.txt
🎁

Liked this tool? Get 50 more AI power moves — free

✓ 50 copy-paste AI prompts
✓ AI & Tech Hacks weekly newsletter
✓ No spam, unsubscribe anytime

Want to know your full AI visibility score?

Robots.txt is just one piece. Get a full audit of how AI search engines see your website, with actionable recommendations.

Get Your Free GEO Scan