Batch scraping API

Batch scraping API: scrape thousands of URLs in one request

Send a list of URLs to POST /v2/batch/scrape and get markdown back for each page. You get a batch id right away, then a webhook or a status endpoint when it is done. No loop, no retry bookkeeping.

Get your API key Batch scrape docs

https://

✓ One request for the whole list
✓ From $0.001 per page
✓ Failed pages are not charged
✓ Cancel anytime

POST /v2/batch/scrape

$ curl -X POST api.webcrawlerapi.com/v2/batch/scrape

→ urls: 3 (1 skipped as invalid)

✓ batch accepted · status_url returned

… scraping 2 / 3

✓ completed · 2 done · 1 failed

GET status_url

{
  "status": "completed",
  "total": 3,
  "completed": 2,
  "failed": 1,
  "canceled": 0,
  "data": [ { "url": "…", "markdown_url": "…" } ]
}

Trusted by 1,000+ developers building AI products and data pipelines

Code

Batch scraping API example: cURL, Python and Node.js

Start the batch, then poll status_url or pass a webhook_url. Skipped URLs come back in invalid_urls.

curl --request POST \
  --url https://api.webcrawlerapi.com/v2/batch/scrape \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "urls": [
      "https://www.example.com/",
      "https://www.example.com/pricing",
      "https://www.example.com/blog"
    ],
    "output_formats": ["markdown"],
    "max_concurrency": 5,
    "webhook_url": "https://your-server.com/webhook"
  }'

Request, then status

// POST /v2/batch/scrape
{
  "id": "6c391693-e566-4b99-97ca-5fa00032e281",
  "status_url": "https://api.webcrawlerapi.com/v2/batch/scrape/6c391693-e566-4b99-97ca-5fa00032e281",
  "invalid_urls": [
    {
      "url": "not a url",
      "error_code": "invalid_request",
      "error_message": "invalid url"
    }
  ]
}

// GET status_url
{
  "id": "6c391693-e566-4b99-97ca-5fa00032e281",
  "status": "completed",
  "total": 3,
  "completed": 2,
  "failed": 1,
  "canceled": 0,
  "data": [
    {
      "url": "https://www.example.com/",
      "status": "done",
      "page_status_code": 200,
      "title": "Example",
      "markdown_url": "https://data.webcrawlerapi.com/.../page.md"
    },
    {
      "url": "https://www.example.com/blog",
      "status": "error",
      "error_code": "webpage_non_success",
      "error_message": "page returned a non-success status code"
    }
  ],
  "next": ""
}

Step by step in the batch scrape guide. Every field is in the API reference.

How it works

From a list of URLs to markdown files

1. Send the list
POST /v2/batch/scrape with urls. The response has the batch id, a status_url and invalid_urls.
2. Wait for it
Set webhook_url for one call at the end, or poll status_url every few seconds. completed, failed and total show progress.
3. Read the items
Each finished item has a markdown_url. Page through results with limit (up to 500) and cursor. Failed URLs: /errors.

Which endpoint

Scrape vs batch scrape vs crawl

The difference is where the URLs come from. Known list: batch. One page: scrape. No list, only a website: crawl.

	Scrape	Batch scrape	Crawl
Input	One URL	A list of URLs you already have	One seed URL, pages are discovered
Endpoint	POST /v2/scrape	POST /v2/batch/scrape	POST /v1/crawl
Result	In the same response	Async: id, status_url, optional webhook	Async: job with a list of pages
Page discovery	None	None, only your URLs	Follows links, sitemaps
Failed pages	success: false in the response	Per item, plus a /errors endpoint	Per item in the job
Cancel a running job	Not needed	DELETE, unstarted pages not charged	Cancel the job
Best for	Single page, agents, quick checks	Sitemap exports, DB lists, search results	Whole sites you do not have a list for

Docs: scrape, batch scrape and crawl API. Single pages first? See the web scraping API.

Features

Built for long URL lists

max_concurrency

Cap how many URLs run in parallel for one batch. Useful when every URL is on the same website.

webhook_url

One POST when the batch finishes, with the same body as the status endpoint (first 100 items).

Errors endpoint

GET /v2/batch/scrape/{id}/errors lists only failed URLs with error codes. robots.txt blocks are listed separately.

Cancel

DELETE a running batch. Items that did not start are marked canceled and are not charged.

Dedup and validation

URLs are normalized and deduplicated. Duplicates and bad URLs come back in invalid_urls, or fail the request with ignore_invalid_urls=false.

Cache with max_age

Default is 7 days, so re-running a list reuses recent results. Set 0 to force fresh pages. Items carry a from_cache flag.

respect_robots_txt

Off by default. Turn it on and blocked URLs are reported under robots_blocked.

Output formats

markdown (default), cleaned, html, links and images. Plus main_content_only and clean_selectors.

prompt, response_schema and actions are not supported in batch scrape. For structured extraction use POST /v2/scrape.

Cost

What a batch costs: 1k, 10k and 100k pages

Pages in a batch are billed at the same per-page rate as a crawl. You only pay for successful pages, and pages that never started in a canceled batch are not charged.

Plan	Per page	1,000 pages	10,000 pages	100,000 pages
Pay as you go	$0.002	$2	$20	$200
Professional ($99/month)	$0.0015	$1.50	$15	$150
Business ($499/month)	$0.001	$1	$10	$100

Standard rate from the pricing docs. Some sites are priced separately, so check the billing page for your exact rate. The number of URLs per batch depends on your plan, so 100,000 pages can mean several batches.

Use case

Sitemap or URL list to a markdown knowledge base

A common job: take every page of a docs site or help center and turn it into clean markdown for your knowledge base or support bot. Get the URL list from a sitemap, send it as a batch, then chunk and embed the markdown.

Sitemap to batch

# 1. Pull URLs from a sitemap (any sitemap parser works)
python sitemap_to_urls.py https://docs.example.com/sitemap.xml > urls.json

# 2. Send them as one batch
curl --request POST \
  --url https://api.webcrawlerapi.com/v2/batch/scrape \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data "{\"urls\": $(cat urls.json), \"main_content_only\": true, \"max_concurrency\": 10}"

# 3. Read markdown_url for every item, chunk it, embed it

Keep it fresh

Run the same list again later. With the default max_age of 7 days, recent pages come from cache; set max_age to 0 when you need everything re-fetched.

No sitemap? Pull links from any page with the URL extractor. Building the whole pipeline? Read website to markdown knowledge base.

Compare

Batch endpoints compared: WebCrawlerAPI, Firecrawl and Olostep

Facts below come from each provider's public docs, read in November 2026. Limits and pricing change, so check the source before you decide.

	Batch endpoint and limits	Notifications	Cancel
WebCrawlerAPI	POST /v2/batch/scrape. Limit is your plan's items limit (100 during trial).	webhook_url, polling with status_url	Yes, with unstarted pages not charged
Firecrawl	/batch/scrape. Sync and async modes, results kept 24 hours after completion.	Webhooks per URL, signed with HMAC-SHA256	Cancel and errors endpoints
Olostep	/v1/batches, up to 100k URLs per batch. New accounts are limited to 100 items per batch.	Async; retrieve content per item	Not covered here

ScrapingBee's HTML API takes one url per request, so a list means your own loop and concurrency control. Looking at other options? Read Firecrawl alternatives or the best web scraping APIs.

Pricing

Simple, transparent pricing

Start with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.

Pay as you go

No monthly commitment — pay only for what you use

$0/month

From $0.002 / page
LLM-cleaned markdown: $0.006 / page
Unlimited proxy included
Up to 5 parallel requests
Pay only for successful requests
Webcrawler agent

Get started

Professional

Best for growing teams

Save 25%

$99/month

From $0.0015 / page
LLM-cleaned markdown: $0.0055 / page
Unlimited proxy included
Up to 20 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Business

For high-volume crawling

Save 50%

$499/month

From $0.001 / page
LLM-cleaned markdown: $0.005 / page
Unlimited proxy included
Up to 50 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Need a custom quota or special pricing? Contact us and we'll set up a plan that fits your needs.

See full pricing, limits and examples →

Batch scraping API FAQ

Common questions about scraping a list of URLs with one request

How many URLs can I send in one batch?

Up to your plan's job items limit per batch. During the trial the limit is 100 URLs. Duplicates are removed and reported in invalid_urls, so they do not count as extra pages.

Should I use a webhook or poll for the result?

Both work. Pass webhook_url and the API sends one POST with the batch status when the batch finishes (first 100 items, the rest through the cursor). Or poll GET /v2/batch/scrape/{id} every few seconds until status is no longer scraping.

How are failed URLs billed?

You only pay for successful requests. Failed URLs are listed with an error code in GET /v2/batch/scrape/{id}/errors, and you can resubmit them in a new batch. If you cancel a running batch, pages that did not start are marked canceled and are not charged.

What happens with duplicate or invalid URLs?

URLs are normalized and deduplicated. Duplicates and rejected URLs come back in invalid_urls with an error code and message. Set ignore_invalid_urls to false if you want the whole request to fail with 400 when any URL is bad.

How much does a batch cost?

Each page is billed at the same per-page rate as a crawled page: $0.002 on Pay as you go, $0.0015 on Professional and $0.001 on Business. That is $2, $1.50 or $1 per 1,000 pages. Some sites are priced separately, see the pricing docs.

Can I limit how fast a batch runs?

Yes. Set max_concurrency to cap how many URLs are scraped in parallel for that batch. This is useful when all URLs are on one website and you want to be gentle on it. 0 or omitted means no per-batch cap.

Does batch scrape support prompt, response_schema or actions?

No. Those parameters are not supported in batch scrape. Use POST /v2/scrape for structured extraction on a single page. Batch supports output_formats, main_content_only, clean_selectors, max_age and respect_robots_txt.

Is the result cached?

Yes. max_age defaults to 604800 seconds (7 days), so a URL you scraped recently can be served from cache. Set max_age to 0 to always fetch fresh content. Each item has a from_cache flag in the status response.

How do I get the content of each page?

Each finished item in the status response has a link: markdown_url, cleaned_url or html_url, depending on output_formats. Download the file from that link. Results are paginated with limit (up to 500) and a cursor.

What is the difference between batch scrape and crawl?

Batch scrape takes an explicit list of URLs and does no discovery. Crawl starts from a seed URL, follows links and decides which pages to fetch. Use batch when you already know the URLs, for example from a sitemap export or your database.

Scrape your first list in five minutes

Get a key, send three URLs with the cURL snippet above and read the markdown links from the status endpoint. Or try a single page first in the playground.

Get your API key