Skip to main content
Paperlink AI can read the public pages of your website. The bot then answers from those pages, and Paperlink AI updates them on a schedule that depends on your plan.

​
What you can import

You can import public pages that need no login. Paperlink AI imports only pages on the same host as the address that you give. It ignores www. when it compares hosts. Paperlink AI also imports only pages under the folder of the address that you give. These examples show the scope.
Paste the folder address that ends in ”/” to limit the import. An address without the final ”/” uses the parent folder.

​
Add a website

1

Open Documents

Select Documents in the dashboard sidebar. Find the Websites card.
2

Start the import

Select Add website.
3

Enter the address

Type the address in Website URL. For example, type https://docs.example.com. Select Find pages.
4

Wait for the page list

The website shows Finding pages while Paperlink AI reads the site. Paperlink AI reads up to 5,000 page addresses for each website.
5

Pick the pages

The page picker shows the sections of your site. Select the sections to import, or select All pages.
6

Save

Select Save. The website shows Importing until the pages are ready.
Each workspace can add a limited number of websites each day. See Automatic updates for the limit. A website that you remove still counts for that day. When you reach the limit, the dashboard shows this message:
You added too many websites today. Try again tomorrow.
When the import ends, the website shows Up to date. These are all the states.

​
How pages count

Each page uses knowledge pages by its text length. This is the same rule as for text files. See Processing and knowledge pages. Imports stop at the knowledge page limit of your plan. The website then shows Partly imported and this message:
Stopped at your knowledge page limit.
The monthly processing limit also applies. When you pass it, the website shows this message:
Monthly processing limit reached. Updates resume next period.

​
Large websites

One update imports pages in steps. A very large website can finish over more than one update. The website then shows this message:
Large site. We import the rest in the next update.

​
Automatic updates

Paperlink AI checks your website again on a schedule. Your plan sets the schedule and the limits. Link depth is the number of link steps from the address that you give. Paperlink AI follows links up to that depth. The trial allows 3 steps and paid plans allow 5. We read up to 5,000 page addresses for each website. Each workspace can add up to 5 websites each day. During the trial, there are no automatic updates. Automatic updates start after the trial ends, if you are on a paid plan. An update skips pages that did not change. A skipped page uses no processing.

​
Update now and Edit pages

Select Update now on a website row to check the site at once. Each plan has a limit of manual updates for each website each day. When you use all of them, the dashboard shows this message:
You used all updates for today. Try again tomorrow.
Open the menu on the row, and then select Edit pages to change the sections that you import. Paperlink AI removes the pages of a section that you clear before it imports the new sections. The first save of the pages for a new website is free. Each later save in Edit pages uses one manual update. Crawled pages belong to their website. They do not show in the documents table. You cannot delete them one by one, and you cannot set an Effective date or a Supersedes link on them. Use Edit pages or Remove instead. An uploaded document can supersede a crawled page. See Versions, effective dates and supersedes.

​
When pages disappear

Paperlink AI removes a page when the page returns “not found” on two updates in a row. A page that is only unreachable for a short time stays. If many pages report “not found” at the same time, Paperlink AI keeps all pages. This applies when more than half of the pages are missing on a website with 4 or more pages. The website then shows Needs attention and this message:
Many pages are missing. We kept your current pages. If the site changed, remove the website and add it again.

​
Sites we cannot import

Paperlink AI cannot import these sites and pages:
  • Pages behind a login.
  • Sites where robots.txt blocks our crawler.
  • Most social networks. Their robots.txt blocks crawlers, or their pages show a login page. A login page can use knowledge pages if the import continues, so use a docs or help center address.
  • Addresses that use an IP address as the host, addresses with a port number, and local or private network addresses.
  • Files that are not HTML. Paperlink AI does not import a PDF that a page links to. Upload the PDF as a document instead.
  • HTML pages over 2 MB.
Some pages need JavaScript to show their text. Paperlink AI renders a limited number of these pages each day. When the daily limit is reached, or a page does not render, the remaining pages wait for the next update. The website then shows Partly imported and this message:
Some pages need a browser to read. We import them in the next update.
When an address is not allowed, the dashboard shows this message:
We can’t import this site. Use a public docs or help center address.

​
Our crawler

Paperlink AI reads your site with this user agent:
If robots.txt has a group for PaperlinkAI-Crawler, Paperlink AI follows only that group. If not, it follows the rules for *. Paperlink AI reads robots.txt again at the start of each update:
  • If robots.txt now blocks a page that you imported, Paperlink AI removes that page. The rule for many missing pages also applies.
  • If robots.txt is not found, Paperlink AI reads all pages in the scope.
  • If Paperlink AI cannot read robots.txt because of a server error or a time-out, it reads no pages and removes no pages. The website shows Partly imported and this message:
We could not reach this site. Check that it is online.
If robots.txt has a Crawl-delay rule, Paperlink AI reads one page at a time and waits between groups of pages. To allow the crawler on your whole site, add these lines to robots.txt:
To block the crawler, add these lines. The website then shows “This site blocks crawlers in robots.txt.”

​
Remove a website

If you remove a website, Paperlink AI deletes its pages from your bot’s knowledge. The bot cannot answer from them again.
1

Open the row menu

Find the website in the Websites card. Open the menu on its row.
2

Remove

Select Remove. Then select Remove again to confirm.
The website shows Removing until the pages are gone.

Processing and knowledge pages

See how Paperlink AI counts knowledge pages.

Upload documents

Add PDFs and other documents by upload.