JavaScript Crawler

JavaScript web crawler for Node.js

Crawl a whole website in one call. Every page comes back as clean markdown, HTML or links. No crawl queue, no Puppeteer, no proxies to maintain. You only pay for pages that succeed.

Get your API key Read the docs

https://

✓ No credit card to try
✓ Failed pages are free
✓ Webhook or polling

One request, whole site

0 / 10 pages

seed URL → follows links → every page back as markdown

job 23b81e21 · running

Trusted by 1,000+ developers building AI products and data pipelines

The problem

Why a hand-rolled Node.js crawler wastes your month

You become the crawl scheduler

A URL queue, a visited set, depth tracking, deduplication and limits. A few lines of cheerio turn into a project.

JavaScript sites hide their links

Client-rendered menus and lists mean a plain fetch finds no links to follow, so your crawl stops at the homepage.

Sites block bots

Rate limits, captchas and blocked IPs mean proxies and retries. That work never ends.

Weighing it up? Read should you build a crawler or use a crawling API.

The output

Pages in, clean markdown out

Menus, banners and footers end up in your embeddings if you do not remove them. Each crawled page comes back as markdown, ready to chunk and embed.

The page: docs.acme.com/api/authentication

Cookie banner: "We use cookies…"

Top navigation: Product · Pricing · Docs · Login

Sidebar menu (40 links)

Authentication

Every request needs an API key in the Authorization header. Create a key in the dashboard…

The part you want

Newsletter signup · Related posts · Footer (60 links)

What the crawl API returns (markdown)

# Authentication

Every request needs an API key in the
Authorization header.

## Create a key

1. Open **Dashboard → Access keys**
2. Click **New key**
3. Copy it. It is shown once.

```bash
curl -H "Authorization: Bearer $KEY" \
  https://api.acme.com/v1/me
```

How it works

How the JavaScript crawler works: one call, every page

1. Send a seed URL
Call client.crawl() with a url and items_limit. Or crawlAsync() to get a job id back right away.
2. The crawler follows links
It discovers pages from the seed in a real browser, staying inside your max_depth, whitelist_regexp, blacklist_regexp and items_limit.
3. Read the pages
crawl() resolves with every page. Each job item has a getContent() method. For big sites use a webhook_url or poll. Read about async requests.

Control what gets crawled

Change the settings and watch the Node.js code update.

items_limit: 100

max_depth

output_formats

items_limit and max_depth cap the size of the job, so a crawl never runs away. You only pay for pages that succeed.

Your Node.js code

import webcrawlerapi from "webcrawlerapi-js";

const client = new webcrawlerapi.WebcrawlerClient("YOUR_API_KEY");

const job = await client.crawl({
    url: "https://docs.acme.com/",
    items_limit: 100,
    output_formats: ["markdown"],
    max_depth: 2,
});

for (const item of job.job_items) {
    console.log(item.original_url);
}

Param	What it does
url	Seed URL where the crawler starts.
items_limit	Stops the crawl when it reaches this number of pages.
max_depth	How many links deep to follow. 0 means the seed page only.
whitelist_regexp	Only URLs matching this pattern are crawled.
blacklist_regexp	URLs matching this pattern are skipped.
respect_robots_txt	Skip pages disallowed by robots.txt. Default false.
output_formats	Any of markdown, html, links. Default markdown.
webhook_url	Receive a POST when the job finishes.

Fresh or cached

Results are cached for 7 days by default. Set max_age to 0 to force a fresh crawl. Details in the caching docs.

Code

How to build a web crawler in JavaScript

import webcrawlerapi from "webcrawlerapi-js";

const client = new webcrawlerapi.WebcrawlerClient("YOUR_API_KEY");

const job = await client.crawl({
    url: "https://docs.acme.com/",
    items_limit: 100,
    max_depth: 2,
    output_formats: ["markdown"],
});

for (const item of job.job_items) {
    const content = await item.getContent();
    console.log(item.original_url, content?.slice(0, 100));
}

The finished job

{
  "id": "23b81e21-c672-4402-a886-303f18de9555",
  "status": "done",
  "url": "https://docs.acme.com/",
  "items_limit": 100,
  "output_formats": ["markdown"],
  "job_items": [
    {
      "original_url": "https://docs.acme.com/api/authentication",
      "page_status_code": 200,
      "status": "done",
      "title": "Authentication | Acme Docs",
      "markdown_content_url": "https://data.webcrawlerapi.com/markdown/..."
    }
  ]
}

See the JavaScript SDK docs. Also available for __SDK__. Need one known page and not a whole site? Use the JavaScript scraper.

Formats

Output formats for every crawled page

markdown

Default. Headings, links and lists are kept. Best for LLM context and knowledge-base chunks.

html

The final HTML of every page, after JavaScript ran.

links

The links found on each page, for building your own sitemap or URL list.

Want every page in one document? Call client.getJobMarkdownContent(jobId).

Use cases

What developers build with the JavaScript crawler

Support bot knowledge base

Before: the bot answers from memory and gets your product wrong.

After: crawl your docs to markdown, then chunk and embed it.

whitelist_regexp: "/docs/"

Help-center sync

Before: articles change and the bot keeps quoting the old ones.

After: re-crawl on a schedule from a Node.js cron job.

max_age: 0

Docs ingestion for AI assistants

Before: an unbounded crawl runs up the bill.

After: cap cost and scope with a page limit and a depth limit.

items_limit + max_depth

Site inventory and link maps

Before: no clear list of what is on the site.

After: get every link on every page for a sitemap or URL audit.

output_formats: ["links"]

Learn more: crawl a documentation site for a knowledge base, keep it in sync, crawling vs scraping.

By the numbers

Reliable infrastructure, proven in production.

Tracking usage, crawl success rates, average response times, uptime percentages, quality scores, cost metrics, API performance, reliability data, and live trends for all your requests.

Active builders

100+

Teams building in production

Extraction quality

98%

Success rate

Fast turnaround

2.5s

Average crawling time

Platform stability

99.98%

Uptime

Which one

Crawlee alternative: WebCrawlerAPI vs Crawlee vs Puppeteer

	WebCrawlerAPI	Crawlee	Puppeteer / Playwright
Finds new pages	Yes, by following links	Yes, with a request queue you run	You write the queue and link logic
Renders JavaScript	Yes	Yes, with a browser crawler (not CheerioCrawler)	Yes
Setup	npm install, one API key	npm install, plus browsers to install and update	Install browsers, manage versions
Runs in serverless	Yes, it is just an HTTP call	Hard with browsers, large binaries	Hard, large binaries
Proxies and retries	Handled for you	Retries built in, you bring the proxies	You build them
Output	Clean markdown, HTML or links	Whatever your handler extracts	Whatever your code extracts
Cost model	Per successful page	Servers, proxies and your time	Servers and your time

Using Python? See the Python web crawler. Comparing providers? Read Firecrawl alternatives and all crawler APIs compared.

Built for sites that fight back

Handled automatically

Residential proxies

Automatic retries

Rate limit handling

Real headless browsers

JavaScript rendering

CAPTCHA solving

Anti-bot bypass

Fastest path selection

Skip browser maintenance. Per-site concurrency is shared and capped, so your crawl will not hammer the target site. See rate limits.

Pricing

Simple, transparent pricing

Start with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.

Pay as you go

No monthly commitment — pay only for what you use

$0/month

From $0.002 / page
LLM-cleaned markdown: $0.006 / page
Unlimited proxy included
Up to 5 parallel requests
Pay only for successful requests
Webcrawler agent

Get started

Starter

Great for small projects

$29/month

From $0.002 / page
LLM-cleaned markdown: $0.006 / page
Unlimited proxy included
Up to 10 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Professional

Best for growing teams

Save 25%

$99/month

From $0.0015 / page
LLM-cleaned markdown: $0.0055 / page
Unlimited proxy included
Up to 20 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Business

For high-volume crawling

Save 50%

$499/month

From $0.001 / page
LLM-cleaned markdown: $0.005 / page
Unlimited proxy included
Up to 50 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Need a custom quota or special pricing? Contact us and we'll set up a plan that fits your needs.

JavaScript Crawler FAQ

Common questions about crawling websites from Node.js

How do I build a web crawler in JavaScript?

Install webcrawlerapi-js, create a WebcrawlerClient with your API key, and call client.crawl({ url, items_limit, output_formats }). The crawler follows links from the seed URL and the result has one job item per page. You write no link queue, no retry logic and no browser code.

What is the difference between crawling and scraping in Node.js?

Scraping fetches the one URL you name (client.scrape()). Crawling starts from a seed URL, discovers pages by following links, and processes many of them (client.crawl()). Use scrape for known pages and crawl for whole sites.

Does it crawl JavaScript-rendered sites?

Yes. Pages are rendered in a real browser before the content and links are extracted, so React, Next.js and other client-rendered sites are crawled like any other.

How do I limit which pages get crawled?

Use whitelist_regexp and blacklist_regexp to include or skip URL patterns, max_depth to limit link depth, items_limit to cap the page count, and respect_robots_txt to follow the site robots.txt rules.

Is crawl() synchronous?

client.crawl() waits until the job finishes and returns all pages. For big sites use crawlAsync(), which returns a job id right away. Then poll getJob(id) or set webhook_url to get a POST when the job is done.

How do I read the crawled content?

Each job item has a getContent() method that returns the content in the format you asked for. To get every page merged into one document, call getJobMarkdownContent(jobId).

Do I need Puppeteer, Playwright or Crawlee?

No. You send a seed URL and get the pages back. Browsers, retries, proxies and anti-bot handling run on our side, so there is nothing to install, patch or scale.

How much does crawling cost?

You pay per successfully crawled page: $0.002 on Starter, $0.0015 on Professional and $0.001 on Business. Failed pages are not charged. Cap the cost of a job with items_limit and max_depth.

Does the crawler respect robots.txt?

Only if you set respect_robots_txt to true. The default is false. Per-site concurrency is shared and capped, so a crawl will not hammer the target website.

Can I try it before paying?

Yes. Use the playground on this page or sign up in the dashboard. No credit card required.

Crawl your first site in five minutes

Get an API key, run the Node.js snippet above, and get markdown for every page of your docs. Or try it in the playground.

Get your API key

JavaScript web crawler for Node.js

Why a hand-rolled Node.js crawler wastes your month

You become the crawl scheduler

JavaScript sites hide their links

Sites block bots

Pages in, clean markdown out

How the JavaScript crawler works: one call, every page

1. Send a seed URL

2. The crawler follows links

3. Read the pages

Control what gets crawled

Fresh or cached

How to build a web crawler in JavaScript

Output formats for every crawled page

markdown

html

links

What developers build with the JavaScript crawler

Support bot knowledge base

Help-center sync

Docs ingestion for AI assistants

Site inventory and link maps

Reliable infrastructure, proven in production.

Crawlee alternative: WebCrawlerAPI vs Crawlee vs Puppeteer

Built for sites that fight back

Simple, transparent pricing

Pay as you go

Starter

Professional

Business

JavaScript Crawler FAQ

Crawl your first site in five minutes