Sitemap mistakes that hide pages from search and AI crawlers

A sitemap can pass every validator and still hide pages. The file is well-formed, the dates are correct, and yet a whole section of the site is missing from it, or the file points crawlers somewhere else. Search engines often find those pages anyway through links. Other crawlers, including some AI crawlers and the tools that import a site into a support chatbot, use the sitemap as their list of pages and may never see them. These are the mistakes that cause it, and a short audit that finds them.

Why the sitemap matters more for some crawlers

A search engine crawls the whole web. It follows links from other sites, keeps a history of your URLs and visits often, so a page missing from your sitemap usually still gets found, only later. A smaller crawler has a fixed budget for your site. Reading the sitemap is the cheapest way to get a page list, so many crawlers start there and some stop there.

Our own crawler is an example. When Paperlink AI imports a docs site or help center, it reads robots.txt, then the sitemaps that robots.txt lists, or /sitemap.xml and /sitemap_index.xml if it lists none. If the sitemap lists pages inside the folder you import, those pages are the import. It follows links only when the sitemap gives it no pages. So a sitemap that leaves out your troubleshooting section means a chatbot that cannot answer troubleshooting questions.

Mistake 1: the sitemap covers only part of the site

This one is easy to miss, and no validator flags it, because the file itself is correct. It happens when a site is built from more than one system:

• The marketing site and blog come from one CMS, and its sitemap plugin lists only those pages.
• The docs or help center runs on another tool, on a subfolder or a subdomain, with its own sitemap or none.
• A new section, such as a changelog or a set of landing pages, is added by hand and never joins the sitemap.

Fix it with a sitemap index that lists every section's sitemap, and list the index in robots.txt. If the help center lives on its own subdomain, give it its own sitemap and its own robots.txt with a Sitemap: line, because each host is read separately.

Mistake 2: robots.txt points to the wrong sitemap

Crawlers that read robots.txt first trust its Sitemap: lines. If a line points to an old domain, a staging host or a file that was renamed, the real sitemap can go unread. Two details make this worse:

• A crawler that finds a Sitemap: line may not also try /sitemap.xml. Ours does not: when robots.txt lists sitemaps, it reads those and skips the usual paths.
• A sitemap on another host is easy to ignore. The sitemaps.org protocol only lets a sitemap list URLs on another host when the robots.txt of that host points to it, and crawlers that do not support this rule skip the file. Our crawler reads only sitemaps on the same host as the site, with or without www..

Check the lines by hand after every domain move or redesign:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap-index.xml
Sitemap: https://example.com/docs/sitemap.xml

Mistake 3: a sitemap in a folder lists pages outside it

The sitemaps.org protocol says a sitemap applies to its own folder and the folders below it. A file at example.com/docs/sitemap.xml may list example.com/docs/... pages, but not example.com/pricing. Crawlers that follow the rule drop the entries outside the folder without telling you. If you need one file for the whole site, put it at the root.

Mistake 4: the sitemap lists pages that robots.txt blocks

A sitemap says "please read this page". A Disallow rule says "do not". When the two disagree, crawlers that respect robots.txt obey the rule and skip the page, so it is as hidden as if it were not listed. This often happens after someone blocks a folder to hide a few test pages and the rule also covers real ones. Test the listed URLs against your rules for each crawler you care about with the robots.txt checker.

Mistake 5: the sitemap lists the wrong version of each page

Google's sitemap documentation asks for the canonical URL of each page: the one address you want shown in results. Sitemaps often list something else instead:

• An address that redirects, for example the old URL of a renamed article.
• An address with tracking parameters or a trailing slash that the page itself does not use.
• A page whose canonical tag points to a different page, or that has a noindex tag.

Each one sends a crawler to a page that says "the real page is elsewhere". A large crawler follows the hint. A small one may spend its budget on redirects and never reach the real page, and a crawler that sees many mixed signals may trust the whole file less. List the final, canonical URL of each page and nothing else.

Mistake 6: crawlers get a different file than you do

You open /sitemap.xml in a browser and see XML. A crawler requests the same address and gets a challenge page, a 403 error or a login page, because a firewall or bot protection treats it as a bot. The sitemap works for every person and fails for every crawler. Test the file from a server, not from your browser. The sitemap checker fetches it from our server and reports when the response is not XML. If the check fails while your browser works, allow the sitemap and robots.txt in your firewall rules for the crawlers you want.

Mistake 7: new pages never join the sitemap

A sitemap that was generated once and saved as a file does not grow. Every page added since then is missing from it. This hurts most on help centers, where new articles answer exactly the questions customers ask now. If you use a static file, regenerate it as part of publishing. Better, let the CMS or the build create it so it changes with the site.

A short audit: compare three lists

Most of these mistakes show up when you compare three lists of URLs for the same site or section:

1. The sitemap list. Run the sitemap checker and note the URL count and any warnings.
2. The link list. Crawl the same start address with the sitemap generator, which follows links and ignores the sitemap. It reads up to 100 pages, so audit one section at a time on a large site.
3. Your own list. Export the published pages from your CMS or help center.

Fix the first row first. A page that is missing from the sitemap is invisible to every crawler that trusts it, while the other rows only slow crawlers down.

After the fixes

Submit the sitemap again in the webmaster tools of each search engine you use, and check the count again a week later. If you import your site into Paperlink AI, use Update now on the website row so the import reads the sitemap again. The website import guide explains how the import reads your site.