This free robots.txt generator creates a complete file for your website in seconds. Choose which AI bots can visit, from ChatGPT and Claude to Perplexity and Google’s AI features, block private folders for every crawler and add your sitemap.
Choose a starting point
CustomFine-tune each bot
Change any switch to make your own set. The starting point then shows as "Custom".
Search and citation bots
5 of 5 allowedLet your pages appear and be cited in AI answers.
- OAI-SearchBot OpenAI
- Claude-SearchBot Anthropic
- PerplexityBot Perplexity
- Googlebot Google
- Bingbot Microsoft
Blocking Googlebot also removes your site from Google Search, not only from AI Overviews.
Blocking Bingbot also removes your site from Bing Search, not only from Copilot.
ChatGPT, Claude and Perplexity will not be able to show or cite your pages.
User-triggered bots
3 of 3 allowedVisit a page when someone asks an AI assistant about it.
- ChatGPT-User OpenAI
- Claude-User Anthropic
- Perplexity-User Perplexity
AI training bots
0 of 7 allowedCollect content to train AI models. Blocking them does not affect AI search.
- GPTBot OpenAI
- ClaudeBot Anthropic
- Google-Extended Google
- Applebot-Extended Apple
- CCBot Common Crawl
- meta-externalagent Meta
- Bytespider ByteDance
Rules for all other bots
These apply to every other crawler, including Google and Bing.
Folders to block
Add a folder below or use Quick add. Until then, every bot can visit every page.
This blocks Google, Bing and every other search engine. Use it only on staging or test sites, never on your live website.
Add your site details (optional)
Paste it to start from your existing rules: we fill in step 3 from it and keep any other rules. Find it at yourwebsite.com/robots.txt.
Your robots.txt
Tip: paste your current robots.txt above so we keep your existing rules.
AI search bots can reach your site. Training bots are blocked.
Where to put it
- In Yoast SEO, go to Tools → File editor. In Rank Math, go to General Settings → Edit robots.txt.
- Replace the contents with the text above and save.
- If you use neither plugin, download the file and upload it as robots.txt to the main folder of your site with your hosting file manager.
- Open yourwebsite.com/robots.txt to check the new rules are live.
- Go to Online Store → Themes, open the menu next to your theme and choose Edit code.
- Under Templates, choose Add a new template, pick robots.txt and create it. Shopify calls the file robots.txt.liquid.
- Paste the Kutola rules (from "# BEGIN Kutola AI bot rules" to "# END Kutola AI bot rules") at the top of robots.txt.liquid, above Shopify's own rules, and save.
- Open yourwebsite.com/robots.txt to check the new rules are live.
- Go to Marketing & SEO → SEO Tools → Robots.txt Editor.
- Replace the contents with the text above and save.
- Open yourwebsite.com/robots.txt to check the new rules are live.
- Squarespace does not let you edit robots.txt.
- To keep AI bots out, go to Settings → Crawlers and turn on "Block known artificial intelligence crawlers". It blocks them as one group, so you cannot choose bots one by one.
- Open yourwebsite.com/robots.txt to check the new rules are live.
- Go to Site settings → SEO → robots.txt.
- Replace the contents with the text above, save and publish your site.
- Open yourwebsite.com/robots.txt to check the new rules are live.
- Download the file.
- Upload it to the main folder (root) of your site, so it opens at yourwebsite.com/robots.txt.
- Open yourwebsite.com/robots.txt to check the new rules are live.
robots.txt is a request that well-behaved bots follow. Firewalls such as Cloudflare can still block bots separately.
Check that it worked
Once your new robots.txt is live, see which AI bots can visit your site.
More free tools
About the generator
What the robots.txt AI Rules Generator does
robots.txt is a short text file at the root of your website that tells crawlers which pages they may visit. The generator writes two kinds of rules for you. First, the rules for 15 AI bots, in three groups:
- Search and citation bots, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, decide whether your pages can appear and be cited in AI answers.
- User-triggered bots, such as ChatGPT-User and Claude-User, visit a page when someone asks an AI assistant about it.
- AI training bots, such as GPTBot, ClaudeBot and Google-Extended, collect content to train AI models. Blocking them does not stop you appearing in AI search.
Second, the rules for every other crawler, including Google and Bing: allow everything, block private folders such as your admin area, cart or search pages, or block the whole site while it is still a staging site.
If you paste your current robots.txt, the generator starts from your existing rules and keeps any it does not manage. Everything happens in your browser: nothing you paste is sent to us or stored.
Frequently asked questions
Is the robots.txt AI Rules Generator free?
Yes. It is free, needs no account and no email address, and works entirely in your browser.
Which folders should I block?
Only areas that should never appear in search, such as an admin area, cart and checkout pages, account pages and internal search results. Do not block your CSS, JavaScript or image folders: search engines and AI systems need them to understand your pages. robots.txt is public, so do not use it to hide private content; use a password or noindex instead.
Which setting should I choose?
For most businesses, “AI search on, training off” is the best starting point. Your pages can still appear and be cited in ChatGPT, Claude, Perplexity and Google’s AI answers, while your content is kept out of future AI training. If you are happy for AI companies to train on your content, choose “Allow all AI bots”.
Will blocking AI bots remove me from Google?
Not with this generator’s presets. Googlebot and Bingbot stay allowed unless you switch them off yourself, because blocking them would also remove your site from normal Google and Bing results. Google-Extended only controls whether Google uses your content for Gemini training and grounding.
Do all AI bots respect robots.txt?
The major AI companies, including OpenAI, Anthropic, Google and Perplexity, say their crawlers follow robots.txt. Some smaller or unknown bots may not. Firewalls and bot protection services, such as Cloudflare, can block bots that ignore the rules.
How long does it take for changes to work?
Bots read robots.txt again regularly, usually within a day, though some may take longer. Once the new file is live, you can check it with our AI Crawler Checker.
I use Squarespace. Can I edit robots.txt?
Squarespace does not let you edit robots.txt directly, but it has a setting to block known AI crawlers. The generator explains where to find it.