> ## Documentation Index > Fetch the complete documentation index at: https://paperlinkai.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Import a website > Add the public pages of your docs or help center to your bot, and keep them up to date. Paperlink AI can read the public pages of your website. The bot then answers from those pages, and Paperlink AI updates them on a schedule that depends on your plan. ## What you can import You can import public pages that need no login. Paperlink AI imports only pages on the same host as the address that you give. It ignores `www.` when it compares hosts. Paperlink AI also imports only pages under the folder of the address that you give. These examples show the scope. | Address that you give | Pages that Paperlink AI imports | | - | - | | `https://acme.com/docs/intro` | Pages under `/docs/` | | `https://acme.com/docs/` | Pages under `/docs/` | | `https://acme.com/docs` | Pages under `/`, which is the whole host | > [!TIP] > > Paste the folder address that ends in "/" to limit the import. An address without the final "/" uses the parent folder. ## Add a website **1. Open Documents** Select **Documents** in the dashboard sidebar. Find the **Websites** card. **2. Start the import** Select **Add website**. **3. Enter the address** Type the address in **Website URL**. For example, type `https://docs.example.com`. Select **Find pages**. **4. Wait for the page list** The website shows **Finding pages** while Paperlink AI reads the site. Paperlink AI reads up to 5,000 page addresses for each website. **5. Pick the pages** The page picker shows the sections of your site. Select the sections to import, or select **All pages**. **6. Save** Select **Save**. The website shows **Importing** until the pages are ready. Each workspace can add a limited number of websites each day. See [Automatic updates](#automatic-updates) for the limit. A website that you remove still counts for that day. When you reach the limit, the dashboard shows this message: > You added too many websites today. Try again tomorrow. When the import ends, the website shows **Up to date**. These are all the states. | State | What it means | | - | - | | Finding pages | Paperlink AI reads the site to find the pages. | | Pick pages | The page list is ready. Pick the pages to import. | | Importing | Paperlink AI reads the pages that you picked. | | Up to date | The pages are imported. | | Partly imported | Some pages are not imported. The row shows the reason. | | Needs attention | An update found a problem. The row shows the reason. | | Failed | The import stopped. The row shows the reason. | | Removing | Paperlink AI removes the website and its pages. | ## How pages count Each page uses knowledge pages by its text length. This is the same rule as for text files. See [Processing and knowledge pages](https://paperlinkai.com/docs/guides/documents/processing.md). Imports stop at the knowledge page limit of your plan. The website then shows **Partly imported** and this message: > Stopped at your knowledge page limit. The monthly processing limit also applies. When you pass it, the website shows this message: > Monthly processing limit reached. Updates resume next period. ## Large websites One update imports pages in steps. A very large website can finish over more than one update. The website then shows this message: > Large site. We import the rest in the next update. ## Automatic updates Paperlink AI checks your website again on a schedule. Your plan sets the schedule and the limits. Link depth is the number of link steps from the address that you give. Paperlink AI follows links up to that depth. The trial allows 3 steps and paid plans allow 5. | Plan | Websites | Automatic updates | Manual updates for each website each day | Link depth | | - | - | - | - | - | | Trial | 1 | None | 1 | 3 | | Starter | 1 | Weekly | 1 | 5 | | Growth | 3 | Daily | 4 | 5 | | Scale | 10 | Daily | 24 | 5 | | Enterprise | Unlimited | Daily | 24 | 5 | We read up to 5,000 page addresses for each website. Each workspace can add up to 5 websites each day. During the trial, there are no automatic updates. Automatic updates start after the trial ends, if you are on a paid plan. An update skips pages that did not change. A skipped page uses no processing. ## Update now and Edit pages Select **Update now** on a website row to check the site at once. Each plan has a limit of manual updates for each website each day. When you use all of them, the dashboard shows this message: > You used all updates for today. Try again tomorrow. Open the menu on the row, and then select **Edit pages** to change the sections that you import. Paperlink AI removes the pages of a section that you clear before it imports the new sections. The first save of the pages for a new website is free. Each later save in **Edit pages** uses one manual update. Crawled pages belong to their website. They do not show in the documents table. You cannot delete them one by one, and you cannot set an **Effective date** or a **Supersedes** link on them. Use **Edit pages** or **Remove** instead. An uploaded document can supersede a crawled page. See [Versions, effective dates and supersedes](https://paperlinkai.com/docs/guides/documents/versions.md). ## When pages disappear Paperlink AI removes a page when the page returns "not found" on two updates in a row. A page that is only unreachable for a short time stays. If many pages report "not found" at the same time, Paperlink AI keeps all pages. This applies when more than half of the pages are missing on a website with 4 or more pages. The website then shows **Needs attention** and this message: > Many pages are missing. We kept your current pages. If the site changed, remove the website and add it again. ## Sites we cannot import Paperlink AI cannot import these sites and pages: * Pages behind a login. * Sites where robots.txt blocks our crawler. * Most social networks. Their robots.txt blocks crawlers, or their pages show a login page. A login page can use knowledge pages if the import continues, so use a docs or help center address. * Addresses that use an IP address as the host, addresses with a port number, and local or private network addresses. * Files that are not HTML. Paperlink AI does not import a PDF that a page links to. Upload the PDF as a document instead. * HTML pages over 2 MB. Some pages need JavaScript to show their text. Paperlink AI renders a limited number of these pages each day. When the daily limit is reached, or a page does not render, the remaining pages wait for the next update. The website then shows **Partly imported** and this message: > Some pages need a browser to read. We import them in the next update. When an address is not allowed, the dashboard shows this message: > We can't import this site. Use a public docs or help center address. ## Our crawler Paperlink AI reads your site with this user agent: ```text theme={"dark"} PaperlinkAI-Crawler/1.0 (+https://paperlinkai.com/docs/guides/documents/website) ``` If robots.txt has a group for `PaperlinkAI-Crawler`, Paperlink AI follows only that group. If not, it follows the rules for `*`. Paperlink AI reads robots.txt again at the start of each update: * If robots.txt now blocks a page that you imported, Paperlink AI removes that page. The rule for many missing pages also applies. * If robots.txt is not found, Paperlink AI reads all pages in the scope. * If Paperlink AI cannot read robots.txt because of a server error or a time-out, it reads no pages and removes no pages. The website shows **Partly imported** and this message: > We could not reach this site. Check that it is online. If robots.txt has a `Crawl-delay` rule, Paperlink AI reads one page at a time and waits between groups of pages. To allow the crawler on your whole site, add these lines to robots.txt: ```text theme={"dark"} User-agent: PaperlinkAI-Crawler Allow: / ``` To block the crawler, add these lines. The website then shows "This site blocks crawlers in robots.txt." ```text theme={"dark"} User-agent: PaperlinkAI-Crawler Disallow: / ``` ## Remove a website > [!WARNING] > > If you remove a website, Paperlink AI deletes its pages from your bot's knowledge. The bot cannot answer from them again. **1. Open the row menu** Find the website in the **Websites** card. Open the menu on its row. **2. Remove** Select **Remove**. Then select **Remove** again to confirm. The website shows **Removing** until the pages are gone. > **[Processing and knowledge pages](https://paperlinkai.com/docs/guides/documents/processing.md)** > > See how Paperlink AI counts knowledge pages. > **[Upload documents](https://paperlinkai.com/docs/guides/documents/upload.md)** > > Add PDFs and other documents by upload. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.