robots.txt generator with AI crawler presets
Pick how AI crawlers should treat your site, add the paths to block and your sitemap. The file updates as you type.
Free, no account needed. Updated .
Short answer
A robots.txt file tells crawlers which paths they may read. This generator writes one from your choices: paths to block, exceptions to allow, your sitemap, and one of three AI presets. Blocking AI training only keeps your pages available to assistants that cite and link sources.
How to use the robots.txt generator
- Choose an AI preset. Allow all, block AI training only, or block all AI crawlers. The section below explains the difference.
- List the paths to block, one per line, such as
/admin/or/search. Add exceptions inside a blocked folder in the second box. - Add your sitemap URL, then copy or download the file and publish it at
https://your-site/robots.txt. - Test it. Paste your site into the robots.txt checker to see what each crawler may read.
The three AI presets
| Preset | Training crawlers | Search and user crawlers | Effect |
|---|---|---|---|
| Allow all AI crawlers | Allowed | Allowed | Assistants can learn from, fetch and cite your pages |
| Block AI training only | Blocked | Allowed | Assistants can still fetch, answer from and link your pages |
| Block all AI crawlers | Blocked | Blocked | Assistants cannot fetch your pages, so they cannot link to you |
The training tokens are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. The search and user tokens are OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Search engines such as Googlebot and Bingbot are never blocked by a preset.
For most businesses with public docs, a help center or product pages, "block AI training only" or "allow all" makes sense. You usually want an assistant to describe your product correctly and send people to your official page. Blocking everything only makes sense for content you sell, or content you do not want quoted at all.
Example output
A store that blocks its admin and cart, allows a help folder inside admin and blocks AI training:
User-agent: *
Allow: /admin/help/
Disallow: /admin/
Disallow: /cart/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /
Sitemap: https://example.com/sitemap.xmlHow the rules are written
- One group for everyone. Your paths go in the
User-agent: *group, which every crawler without its own group follows. - Allow before Disallow. Order does not change the result under RFC 9309 (the longest matching rule wins), but listing exceptions first is easier to read.
- One group for blocked AI crawlers. Several
User-agentlines can share one set of rules. A crawler that has its own group ignores the*group, which is why the blocked AI group simply saysDisallow: /. - Sitemap lines at the end. They apply to the whole file wherever they are, and must be full URLs.
- Crawl-delay is optional. Bing and some other crawlers slow down when it is set. Google ignores it. The generator caps it at 60 seconds.
What to block, and what not to
Good candidates for Disallow:
- Admin and account areas, carts and checkout steps.
- Internal search result pages, which create endless URLs.
- Filter and sort parameters, with a wildcard such as
/*?sort=.
Do not block:
- CSS, JavaScript and image folders that your pages need to render.
- Pages you want out of search results: use a noindex tag instead, which crawlers must be able to read.
- Your help center or docs, if you want assistants and support chatbots to answer from them.
Staging sites
The "block the whole site" option writes Disallow: / for everyone. Use it on staging and test hosts only, and add a password as well: robots.txt is a request, not a lock. Before launch, check that the production file does not contain this line. It is one of the most common reasons a new site never appears in search.
After you publish
Crawlers cache robots.txt, often for up to a day, so a change is not applied at once. Open the live file in a browser to confirm it loads as plain text with a 200 status, then run the checker. If you changed AI rules, ask an assistant about one of your pages a few days later to see whether it can still read it.
When the file is live, check the sitemap it points to with the sitemap checker.
Frequently asked questions
What should a basic robots.txt contain?
A "User-agent: *" group with the paths you do not want crawled (admin, cart, internal search results), and a "Sitemap:" line with the full URL of your sitemap. If you block nothing, "Allow: /" is enough.
What is the difference between blocking AI training and blocking AI search?
Training crawlers such as GPTBot, ClaudeBot and Google-Extended collect text to train models. Search and user crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so an assistant can answer and link to them. Blocking training keeps you citable; blocking search does not.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended is a control token for Gemini training and grounding. Googlebot, which crawls for Google Search, is a different token and is not affected.
Can robots.txt protect private pages?
No. robots.txt is public and only asks well-behaved crawlers to stay out. Anyone can read it and open the paths it lists. Protect private pages with a login.
Where do I upload robots.txt?
To the root of the host, so it loads at https://example.com/robots.txt. Each subdomain needs its own file. In most CMSs you edit it in the SEO settings; static sites keep it in the public folder.
Newsletter
Support notes, once a month.
One email with what we learn about help centers, honest AI answers and support teams. Unsubscribe at any time.