See which AI bots can visit your website, including the ones behind ChatGPT, Claude, Perplexity and Google’s AI features. Enter any page and get the answer in seconds, with the exact robots.txt lines to fix anything that is blocked.
About the checker
What the AI Crawler Checker looks at
The checker reads your website’s robots.txt file and works out, for the page you entered, whether each of 15 AI bots is allowed to visit it. It also checks the page itself for settings that tell search engines and AI systems not to show it, such as noindex.
The bots are grouped by what they do:
- Search and citation bots, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, decide whether your page can appear and be cited in AI answers. These matter most.
- User-triggered bots, such as ChatGPT-User and Claude-User, visit a page when someone asks an AI assistant about it.
- AI training bots, such as GPTBot, ClaudeBot and Google-Extended, collect content to train AI models. Blocking them is your choice and does not stop you appearing in AI search.
What you get
A clear verdict, the number of AI bots that can visit the page and a full table of all 15. If a search or user-triggered bot is blocked, you get the rule that blocks it and ready-made robots.txt lines to let it in. There is no score here: to see how well AI systems can read and cite the page once they reach it, run the full AI Readiness Checker.
Frequently asked questions
Is the AI Crawler Checker free?
Yes. It is free to use and does not need an account or an email address.
Should I block AI training bots?
That is a business decision. Blocking training bots such as GPTBot or ClaudeBot keeps your content out of future AI training, but it does not stop your pages appearing in ChatGPT, Claude or Perplexity answers, because those use separate search bots. The checker shows training bots for information and never treats blocking them as a problem.
Which bots matter most for AI search?
The search and citation bots: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, and Googlebot and Bingbot, which also feed Google’s AI Overviews and Microsoft Copilot. If these are blocked, your page is unlikely to be cited in AI answers.
My robots.txt allows every bot. Can AI bots still be blocked?
Yes. Firewalls and bot protection services, such as Cloudflare, can block AI bots no matter what your robots.txt says. Those settings cannot be tested from outside your website, so check your firewall or bot protection settings as well.
Why does the result depend on the page I enter?
robots.txt rules can allow or block bots for different parts of a website, for example only for your /blog/ pages. The checker applies the rules to the exact page you entered, so it is worth checking your homepage and your key service or product pages.
How do I change my robots.txt?
robots.txt is a plain text file at the root of your website, for example yourwebsite.com/robots.txt. On WordPress you can usually edit it with your SEO plugin, such as Yoast SEO or Rank Math. On Shopify, Wix and Squarespace it is in the site’s SEO settings. Copy the lines the checker gives you, add them, then run the check again.