Free website crawler and link extractor

How to use the website crawler

1. Enter a start address. Use your home page to crawl the whole site, or a folder such as example.com/help/ to crawl one section. The crawler stays on that host and under that path.
2. Wait for the crawl. It reads up to 100 pages, six at a time, following the links it finds. A slow site can take up to a minute.
3. Read the summary. It counts pages that work, pages that redirect, broken pages (4xx), server errors (5xx) and pages that did not load.
4. Filter and export. Tick "Show only redirects and errors" to see what needs fixing, then download the CSV to share it or sort it in a spreadsheet.

What the crawl report shows

The CSV has the same columns plus the content type and any error. Values that start with characters a spreadsheet reads as a formula get an apostrophe in front, so a page title can never run as a formula.

A crawl report, not a sitemap

The sitemap generator uses the same kind of crawl, but its job is a clean list of pages to submit to search engines: it leaves out errors and follows canonical tags. This crawler does the opposite. It keeps every page it read, broken and redirected pages included, because those are the pages a link audit is for. It does not follow canonical tags, so you see each address as visitors reach it.

What the crawler reads and skips

• robots.txt: pages that robots.txt blocks for our crawler are not read, and the summary counts the links it left out. If robots.txt blocks the start page, nothing is read.
• Same host and path only: links to other sites are counted but not followed. The host with and without www counts as the same site.
• Links, not scripts: the crawler reads the HTML the server sends. It does not run JavaScript, so links that only appear after scripts run are not found. Links marked rel="nofollow" are not counted or followed.
• No files or traps: images, PDFs, scripts and fonts are skipped, as are session URLs and endless calendar or paging links.
• Limits: 100 pages read, 5 clicks deep. When it stops at a limit, the report says how many more URLs it found but did not read.

Run a link audit in four steps

1. Fix broken links first. Each 4xx page shows the page that links to it. Change the link to the right address, or redirect the old address if other sites link to it.
2. Shorten redirects. A page that redirects costs visitors an extra round trip. Update internal links to point straight to the final address. Check long chains with the redirect checker.
3. Look at depth. Important pages more than three clicks from the start page are hard to find for people and for crawlers. Link to them from your navigation or a hub page.
4. Check titles. Empty or repeated titles make pages hard to tell apart in search results and in AI answers. Sort the CSV by title to find duplicates.

Extract links from a website

If you need the list of addresses on a site, download the CSV and use the URL column: it holds every page the crawler read, in the order it found them. The link counts show which pages link out the most, which is useful when you check outbound links before a redesign or a domain move.

Why this matters for AI assistants and chatbots

AI assistants and support chatbots that import a website reach pages the same way: by following links from pages they already know. A help article that no page links to, or one that only loads with JavaScript, may never be read. A broken link in your help center is a dead end for a crawler as much as for a customer. The AI readiness grader checks the other half: robots.txt, sitemap and page text.

Limits of a free crawl

One run reads 100 pages, which covers a small site or one section of a larger one. Crawl sections one at a time for a bigger site. Results are cached for one hour per address, so a second run within the hour shows the same report.