How do you crawl a documentation site for an LLM knowledge base?

Answer

Crawl a documentation site by scoping the crawl to the docs subtree with a URL filter, deciding between sitemap.xml and link discovery, stripping the nav and sidebar so the text is only the page's own content, and taking markdown out the other end. Then pin one doc version, keep concurrency low, and schedule a re-crawl so the index does not go stale. The order matters: everything you get wrong at crawl time turns into a bad chunk, and a bad chunk is a wrong answer your support bot gives with total confidence.

Scope the crawl to the docs subtree

A documentation site is almost never a standalone domain. example.com also carries a marketing site, a pricing page, and a blog. Seed the crawler at the root and most of what you pay for is content you do not want in a knowledge base — and marketing copy is the worst possible neighbour for a technical answer, being confident, keyword-dense, and wrong about specifics.

Start the crawl at the docs root and constrain it with a URL pattern. Two filters do the work:

• A whitelist pattern keeps the crawler inside /docs/.
• A blacklist pattern drops the parts of the subtree that are noise: changelogs, API reference pages that are pure generated tables, /docs/search, tag pages.

If you do not know the URL structure yet — and you usually do not — run a small unfiltered crawl first with a low page limit, then look at the URLs it found. GET /v1/job/:id/urls returns the discovered URLs grouped into path clusters, so you can see the shape of the site (/docs 412 pages, /blog 87) before committing to a pattern. Write the regex against that list, not a guess.

Sitemap.xml vs link discovery

Both work. They fail differently.

Sitemap.xml wins when the site publishes a complete one. You get the full page list in a single request, including orphan pages that nothing links to, and you get lastmod timestamps that tell you what changed since your last run. Docusaurus, MkDocs, GitBook, and Mintlify all generate sitemaps by default, so for a typical modern docs site the sitemap is right there at /sitemap.xml. Fetch it, filter the URL list yourself, and scrape each URL directly with POST /v2/scrape. You control exactly which pages get hit.

Link discovery wins when the sitemap is missing, stale, or truncated — common on self-hosted docs and on sites whose sitemap only lists the latest version. It also wins when you do not want to maintain a URL list: point a crawler at the docs root, let it follow links, and new pages appear in the next run without you touching anything. This is what POST /v1/crawl does, with max_depth to bound how far from the seed it travels.

The practical answer for most teams: check for a sitemap and use it to sanity-check the crawl rather than replace it. If the crawler found 340 pages and the sitemap lists 412, you have a discovery gap worth understanding before you ship the index.

Strip the boilerplate before it becomes a chunk

This is the step people skip, and it is the one that quietly ruins retrieval quality.

Every page on a docs site carries the same sidebar, top nav, footer, "Was this page helpful?" widget, and often the same right-hand table of contents. Keep that markup and every chunk contains the same few hundred repeated words. Two things go wrong. Short pages become almost entirely boilerplate, so their embeddings cluster by navigation rather than by topic. And a chunk that straddles the sidebar returns a list of link labels as an "answer."

Two levers handle it:

• main_content_only: true targets the primary content region and drops navigation, sidebars, and ads.
• clean_selectors takes a comma-separated list of CSS selectors to remove. Use it for the site-specific leftovers main_content_only does not know about — a docs-specific .theme-doc-toc-desktop, a .feedback-widget, a version-picker dropdown.

One thing to know about clean_selectors: it replaces the default removal list rather than adding to it. The default is script, style, noscript, iframe, img, footer, header, nav, head. If you pass only .feedback-widget, you have just turned footer and nav stripping back on. Always include the defaults alongside your own selectors.

Markdown is the output format you want

Ask for markdown. Headings survive as ##, code blocks survive as fenced blocks, lists stay lists. That structure is what a header-aware chunker splits on, so chunks land on real section boundaries instead of arbitrary character counts. It also keeps code samples intact, which matters enormously for technical docs — a truncated code block is worse than no code block.

Plain text throws away the heading hierarchy. Raw HTML forces you to strip tags anyway and burns tokens on markup.

Handling versioned documentation

Versioned docs are the single most common source of confidently wrong answers. If /docs/v1/auth and /docs/v2/auth are both in your index, the retriever will happily hand back the v1 method signature for a v2 question, and nothing downstream flags it.

Pick one canonical version and enforce it with URL filters. If the site publishes /docs/latest/, whitelist that and blacklist the numbered paths. If it uses /docs/v2/ as current, whitelist v2 explicitly — latest aliases are convenient but they shift under you, and a re-crawl six months later silently swaps your entire index to a new major version.

If you genuinely need multiple versions indexed, crawl them as separate jobs and carry the version as metadata on every chunk, so you can filter at query time. Do not mix them into one flat namespace.

Rate limits and politeness

Docs sites are usually small, often static-hosted, and rarely provisioned for a crawler hitting them at full speed. Keep concurrency modest and the crawl still finishes fine — a few hundred pages is not a large job. WebCrawlerAPI caps concurrency at 10 parallel threads per account, and separately at 10 parallel threads per target website shared across all accounts, so you cannot accidentally hammer a host even if you try. Set respect_robots_txt: true as a default. Docs sites sometimes disallow search and print-view URLs precisely because they are duplicate content, and honouring that gets you a cleaner index for free.

Re-crawling for freshness

Documentation changes. A knowledge base built once is accurate for about a quarter.

Two approaches. Re-run the same crawl job on a schedule — weekly is reasonable for active products — and re-embed pages whose content hash changed. Or use a feed: POST /v1/feeds monitors a URL, crawls it periodically, and delivers only what changed via webhook, RSS, or JSON. It takes the same whitelist_regexp, blacklist_regexp, main_content_only, and max_depth parameters as a crawl job, so the scoping work you already did carries straight over. The webhook route is the one you want if you are updating a vector index — you get told what changed instead of diffing the whole site yourself.

A worked example

Crawling the v2 docs of an imaginary product, skipping the changelog and API reference dumps, with clean markdown out:

curl --request POST \
  --url https://api.webcrawlerapi.com/v1/crawl \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://example.com/docs/v2/",
    "items_limit": 500,
    "output_formats": ["markdown"],
    "whitelist_regexp": "/docs/v2/.*",
    "blacklist_regexp": "/docs/v2/(changelog|api-reference)/.*",
    "main_content_only": true,
    "clean_selectors": "script, style, noscript, iframe, img, footer, header, nav, head, .theme-doc-toc-desktop, .feedback-widget",
    "respect_robots_txt": true,
    "webhook_url": "https://yourserver.com/webhook"
  }'

Reading it back: the crawl starts at the v2 docs root, never leaves /docs/v2/, drops changelog and generated reference pages, returns markdown with the article body only, keeps the default tag stripping while also removing this site's table of contents and feedback widget, honours robots.txt, and posts to your webhook when the job finishes. Crawling is asynchronous — the immediate response is a job id, and each finished page carries a markdown_content_url you fetch and hand to your chunker.

Where WebCrawlerAPI fits

You can build all of this yourself with a headless browser and a queue. Most teams do, once, and then spend the next two months on retries, blocked requests, JavaScript-rendered docs, and boilerplate stripping. WebCrawlerAPI gives you the crawl, the URL filtering, the cleaning, and the markdown in one API call, plus scheduled feeds when you need the index to stay current.

If you are building a knowledge base or a support bot on top of somebody's documentation, start with WebCrawlerAPI — point it at a docs root and see what the markdown looks like before you commit to a pipeline.