1 · Queue
The crawler keeps a first-in, first-out list of URLs to visit. It starts with one seed URL and takes the next one each round.
urlInteractive demo
How a web crawler works
Start from one URL. Scrape it, read every link on the page, queue the ones on the same site and record the ones that point elsewhere. Repeat until the queue is empty. This demo replays a crawl over a sample site, so any domain you type gets the same example pages.
1 · queueTake next URL
The queue starts with the seed URL.
2 · scrapeFetch the page
One scrape() call returns HTML, markdown and links.
3 · extractRead every <a href>
Relative links become absolute URLs.
4 · filterQueue or record
Same site → queue. Other site → external_links.
Crawl map
drag nodes to rearrangecrawled 0/20queue 1external 0
Scroll sideways to explore the map.
- Crawled page
- In queue
- Scraping now
- External link (not followed)
- Skipped by max_depth / items_limit
Queue
Crawler log
seed https://example.com/ · max_depth 2 · items_limit 20
Each page in the crawl is one SDK call
import webcrawlerapi from "webcrawlerapi-js"
const client = new webcrawlerapi.WebcrawlerClient(process.env.WEBCRAWLER_API_KEY)
const page = await client.scrape({
url: "https://example.com/",
output_formats: ["markdown", "html", "links", "external_links"],
})
page.markdown // clean markdown for your model
page.raw_content // original HTML
page.links // 9 internal + 2 external URLsThe loop
The crawl loop, step by step
2 · Scrape
It fetches the page and gets back HTML, markdown and the links on it. A failed page is retried or marked failed, and the crawl moves on.
output_formats3 · Extract
It reads every <a href>, turns relative links into absolute URLs and removes duplicates.
links4 · Filter
Same-site links that were not seen yet go to the queue. Other sites are only recorded. Pages past the limits are skipped.
max_depth, items_limitLinks
Internal vs external links
A link to the same site (ignoring www.) is internal. The crawler follows it. A link to another site is external. The crawler never fetches it, but with external_links on it still lists the URL in links. In the map above, internal pages are circles and external links are pink diamonds.
Filter
Which links a crawler ignores
| Link | Why |
|---|---|
| mailto:, tel:, javascript: | Not pages. Nothing to fetch. |
| #anchors | Same page, different scroll position. |
| /static/, /assets/, /_next/, /cdn-cgi/ | Site assets, not content. |
| Media files | Images, video and archives are not crawled as pages. |
Limits
Limits: max_depth and items_limit
Without limits a crawl of a big site never ends and costs grow with every page. max_depth stops the crawler a fixed number of clicks from the start page. items_limit stops it after a number of pages. Try max_depth 3 or items_limit 8 in the demo above and watch which pages are skipped.
Output
From crawled page to markdown for your knowledge base
Open the Markdown and HTML tabs of any crawled page in the demo. The markdown is a fraction of the HTML and keeps headings, lists and links, so it chunks cleanly for a knowledge base. See markdown cleaning for how menus and footers are removed.
Code
Run the same crawl with one request
import webcrawlerapi from "webcrawlerapi-js";
const client = new webcrawlerapi.WebcrawlerClient("YOUR_API_KEY");
const page = await client.scrape({
url: "https://example.com/blog",
output_formats: ["markdown", "links"],
});
console.log(page.markdown);
console.log(page.links);More in the scrape API, crawl API docs and on the Crawl API page. The JavaScript SDK is on npm.
Web crawler FAQ
Quick answers about how crawling works
What is a web crawler?
A web crawler is a program that starts from one URL, fetches the page, reads the links on it and repeats the process for the new pages it finds. It keeps going until the queue of URLs is empty or a limit is reached.
How does a web crawler find new pages?
By reading every <a href> on each page it fetches. Relative links become absolute URLs, and links to the same site that were not seen before are added to the queue.
What is the difference between crawling and scraping?
Scraping fetches one URL you name and returns its content. Crawling discovers pages on its own from a seed URL and scrapes many of them. In WebCrawlerAPI, scrape is /v2/scrape and crawl is /v1/crawl.
Does a crawler follow external links?
No. Links to other sites are recorded but never fetched. With external_links enabled on a scrape request, they are returned in the links list so you can see where a page points.
Which links does a crawler skip?
mailto: and tel: links, same-page #anchors, javascript: links, static files such as /static/ or /assets/ paths and media files. These are not pages, so there is nothing to crawl.
What do max_depth and items_limit do?
max_depth limits how many clicks away from the start page the crawler goes. items_limit stops the crawl after that many pages. Both keep cost and crawl time under control.
Does the crawler respect robots.txt?
Only if you set respect_robots_txt to true. The default is false. Per-site concurrency is shared and capped, so a crawl will not hammer the target website.
Why convert crawled pages to markdown?
Markdown keeps headings, lists and links and drops the markup, so it takes fewer tokens and chunks cleanly. That makes it a good input for a knowledge base or support bot.