Should you block GPTBot? What blocking AI crawlers really does
GPTBot is only one of several AI crawlers. What blocking training, search and user crawlers each does, what it does not do, and how to choose.
By the Paperlink AI teamPublished 6 min read
Short answer
Blocking GPTBot asks the OpenAI training crawler to stop collecting your pages from now on, and nothing more: ChatGPT search uses a different crawler, OAI-SearchBot. Most businesses with public product or help pages should keep search and user crawlers allowed, so assistants can cite and link them, and decide about training crawlers separately.
Many sites added User-agent: GPTBot and Disallow: / to robots.txt after it became known that AI companies train models on public web pages. Some of those sites wanted to stop training. Many also stopped AI assistants from reading and linking their pages, without meaning to. The cause is that each AI company runs several crawlers for different jobs, and one line in robots.txt controls only one of them. This post explains the crawlers, what blocking each one really does, and how to choose.
AI crawlers do three different jobs
- Training crawlers collect public pages that may be used to train future AI models. Blocking them affects what future models learn from your site.
- Search crawlers build the index that an assistant searches when it answers a question and links its sources. Blocking them affects whether an assistant can find and cite your page today.
- User-initiated fetchers open a page because a person asked the assistant to, for example by pasting your link into a chat. Blocking them affects whether the assistant can read the page that the person asked about.
The main vendors publish a separate token for each job:
| Vendor | Training | Search | User-initiated |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed | PerplexityBot | Perplexity-User |
| Google-Extended (a control token) | Googlebot (Google Search) | None listed | |
| Apple | Applebot-Extended (a control token) | Applebot | None listed |
| Meta | meta-externalagent | None listed | meta-externalfetcher |
| Common Crawl | CCBot (an open archive that many models train on) | None | None |
Google-Extended and Applebot-Extended are not separate crawlers. They are tokens that you name in robots.txt to say whether pages that Googlebot or Applebot already fetched may be used for those companies' AI models. Vendors add and rename tokens, so check each vendor's crawler page before you rely on this list.
What blocking GPTBot does
It asks OpenAI's training crawler not to collect the paths you block, from the time it next reads your robots.txt. That is all. OpenAI documents that GPTBot follows robots.txt, so for that one crawler and that one job, the rule works.
What it does not do
- It does not remove what was already collected. robots.txt works from now on. Pages collected before the rule are not deleted by it.
- It does not block ChatGPT search. That is OAI-SearchBot, a different token. If you block only GPTBot, ChatGPT can still find and link your pages in search answers.
- It does not block other companies. Each vendor reads only the group that names its own token, or the
*group when none does. - It does not block copies. If your text is in a public archive such as Common Crawl, or quoted on other sites, it can reach a model from there. CCBot is a separate token.
- It does not enforce anything. robots.txt is a request. Well-known crawlers follow it; a scraper that ignores it is not stopped. Content that must stay private needs a login.
- It does not change your Google ranking. GPTBot has nothing to do with Google. And blocking Google-Extended does not remove you from Google Search, because Googlebot is the token for Search.
What blocking search and user crawlers costs
If you block the search and user-initiated tokens too, an assistant cannot fetch your pages. It cannot quote your current prices, link your help article or read the page a user pasted. The assistant may still answer questions about you from other sources: reviews, forum posts, old copies of your pages. Those answers can be out of date or wrong, and the person who asked gets no link to your official page.
For most businesses, that is the opposite of what they want. If someone asks an assistant how your returns work, the best answer comes from your returns page, with a link to it.
How to decide, by type of site
| Type of site | Training crawlers | Search and user crawlers | Why |
|---|---|---|---|
| Software product, docs or help center | Your choice | Allow | You want assistants to describe the product correctly and link the official page |
| Online shop or local business | Your choice | Allow | Prices, opening hours and policies are what people ask assistants about |
| Publisher whose articles are the product | Often blocked | Decide per section | Free summaries can replace a visit; links can also bring readers |
| Paid courses, research or other content you sell | Block | Block on the paid paths | The content itself is what you charge for |
| Staging or test sites | Block | Block | Block every crawler, and add a password |
How to write the rules
A common middle ground: keep the whole site open to search and user crawlers, block training crawlers on the paid section only, and keep a private area closed to everyone. For a site that sells courses under /courses/:
User-agent: *
Disallow: /account/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /account/
Disallow: /courses/
Sitemap: https://example.com/sitemap.xmlNote the second Disallow: /account/. A crawler that finds a group with its own token follows only that group and ignores the * group. If you leave the line out, you block training on /courses/ but open /account/ to the training crawlers. Repeat every general rule in each named group.
You do not have to write this by hand. The robots.txt generator has presets for allowing all AI crawlers, blocking training only, or blocking all of them, and it adds your own paths and sitemap.
Check the result
After you publish the file, test it with the robots.txt checker. Enter your site and the path you care about, such as /courses/ or /help/, and read the result for each crawler. Check two things: the training tokens show the result you chose, and the search tokens are still allowed on your help and product pages. Crawlers cache robots.txt, often for up to a day, so a change is not applied at once.
What about support chatbots that import your site?
A support chatbot that imports your help center uses its own crawler token, so a list of AI training tokens does not affect it. Paperlink AI's crawler is PaperlinkAI-Crawler. It follows a group that names it, or the * group when there is none, and it reads robots.txt again at the start of each update. A rule that blocks it, or a Disallow: / for *, stops the import. The website import guide has the lines to allow or block it. On paperlinkai.com itself, we allow every named AI crawler on our public pages, because we want assistants to describe the product from our own pages.
Questions people ask
If I block GPTBot today, will ChatGPT forget my site?
No. Blocking stops future collection by that crawler. It does not remove anything already collected, and it does not stop ChatGPT search, which uses OAI-SearchBot.
Do all AI crawlers follow robots.txt?
The vendors in the table above document that their crawlers read robots.txt. Some say that fetches a user asks for directly can be treated differently, so read each vendor's crawler page for the current rules. Crawlers that do not identify themselves cannot be controlled with robots.txt at all.
Should I block AI crawlers to protect my server?
If crawlers cause real load, slow them down with rate limits at your host or CDN rather than blocking them in robots.txt. A block also removes the benefit of being read and cited.