Free robots.txt checker and tester

How to use the robots.txt checker

1. Enter your website. Type the domain, such as example.com. The checker reads https://example.com/robots.txt.
2. Add a path to test (optional). Enter a path such as /docs/ or paste a full URL on the same site. Leave it empty to test only the home page.
3. Read the table, then the line notes. The table shows, for each crawler, whether it may read the home page and your path. The notes under it point to the lines that need a fix.

What the checker tests

The checker fetches robots.txt the way a crawler does and applies the matching rules of RFC 9309, the robots exclusion standard. It uses the same matcher as the Paperlink AI crawler. For each crawler it finds the group that names that crawler. If there is none, it uses the * group. Then it picks the longest rule that matches the path.

It also lints each line: rules that come before any User-agent line, misspelled fields, paths that do not start with / or *, Sitemap lines without a full URL, and a Noindex line, which Google has ignored since 2019.

What the result means

Allowed means the crawler may fetch that path. Blocked means a rule stops it. "Own group" next to a crawler means robots.txt has a group just for it, so the * rules do not apply to it at all. That is the most common surprise: a site that blocks /private/ for * and then adds a GPTBot group with other rules has unblocked /private/ for GPTBot.

The status of the file matters as much as its content. A 404 means "no rules", so everything is allowed. A server error or a time-out means "stop": most crawlers read nothing until robots.txt loads again. A broken robots.txt can stop crawling of the whole site.

robots.txt syntax in one minute

A file is a list of groups. Each group starts with one or more User-agent lines, followed by rules:

User-agent: *
Disallow: /admin/
Disallow: /*?sessionid=
Allow: /admin/help/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

• * in a path matches any run of characters. $ at the end anchors the match to the end of the URL, so /*.pdf$ matches every PDF.
• Paths are case-sensitive. User-agent tokens are not.
• An empty Disallow: allows everything. Disallow: / blocks everything.
• Sitemap lines apply to the whole file, wherever they appear.

Should you block AI crawlers?

It depends on what you want from AI assistants. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect text for future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so that an assistant can cite and link them in an answer today. Blocking the search crawlers means assistants can't link to you, even when your page is the right answer.

Many businesses with public docs or help centers allow both: they want assistants to answer questions about their product correctly and to send people to the official page. If you block training but allow search, give each token its own group and test the result here.

Common robots.txt mistakes

• Blocking CSS and JavaScript. Search engines render pages. If they cannot load your scripts and styles, they may misread the page.
• Using robots.txt to hide a page. A blocked URL can still appear in search results from links. Use a noindex tag, or a login for private content.
• A staging file in production. Disallow: / copied from a staging site is a classic launch-day error. The checker flags a * group that blocks the whole site.
• Forgetting subdomains. docs.example.com and help.example.com each need their own robots.txt.

When your robots.txt is clean, check the sitemap it points to with the sitemap checker.