AI crawler robots.txt checker and generator
Make a robots.txt
Tick the crawlers to block. Every other crawler, and the search engines, stay allowed.
How to use it
- Paste your robots.txt (open yoursite/robots.txt in your browser and copy it).
- Type a path such as /blog/ if you want to know what happens there too, then press Check the crawlers.
- Read the table: Allowed, Blocked or Not mentioned for each crawler, with the line that decided it and the User-agent group it came from.
- To write a new file, tick the crawlers to block, add your sitemap address, press Make robots.txt and then Copy.
Limits
- A crawler that ignores robots.txt is not stopped by it: the file is a request, which the well-known crawlers honour.
- The check reads only what you paste; it does not fetch your site, so paste the file your server really serves.
- A line the standard does not define (a path with no leading slash, a rule before any User-agent) is read the way the common crawlers read it, which an unusual crawler may not match.
Questions
Which AI crawlers should I block?
Your choice. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps new models from learning from your pages; blocking the search and user-fetch crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) also keeps your pages out of the answers people ask for. This site allows them all.
Does a Disallow under User-agent: * block the AI crawlers too?
Only a crawler with no group of its own. A crawler named in its own User-agent group follows that group alone and ignores the * group, so a Disallow under * does nothing for GPTBot when a GPTBot group exists.
Why does it say Allowed when there is no rule?
A group with no matching Allow or Disallow line permits the path: robots.txt blocks only what it names. Not mentioned means the file has no group for that crawler and no * group either, which also allows it.
Is what I paste sent anywhere?
No. The check runs in your browser; nothing is uploaded and the page keeps no copy. The site counts that the tool was used (the page, the event and the site that linked you), with no cookie and no id.
Sources
- The site’s own robots.txt and the AI crawlers it names
tools/website/framer/build.mjs - The tool’s logic and its tests
tools/website/learn/tools/lib/robots.mjs