WebCrawlerAPI
Guides17API reference20SDKs4Integrations7

Guides/Markdown & scrape/Batch scrape

Batch scrape

Scrape a list of URLs in one async request and collect results

Open in Claude Open in ChatGPT

Batch scrape is for when you already know the URLs: a sitemap export, a list from your database, search results. You send the list once and the API scrapes them in the background. No crawling, no per-URL requests.

For single pages use Scrape. To discover pages on a website use Crawl.

1. Start a batch

bash
curl --request POST \
  --url https://api.webcrawlerapi.com/v2/batch/scrape \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "urls": [
      "https://www.example.com/",
      "https://www.example.com/pricing",
      "https://www.example.com/docs"
    ],
    "output_formats": ["markdown"],
    "main_content_only": true
  }'

The response has the batch id, a status_url, and invalid_urls for anything skipped (bad URL, duplicate, blocked domain):

json
{
    "id": "6c391693-e566-4b99-97ca-5fa00032e281",
    "status_url": "https://api.webcrawlerapi.com/v2/batch/scrape/6c391693-e566-4b99-97ca-5fa00032e281",
    "invalid_urls": []
}

Limits: up to your plan's items limit per batch (100 URLs during trial). Set ignore_invalid_urls to false if you want the whole request to fail when one URL is bad.

2. Wait for the result

Pick one:

Webhook. Pass webhook_url in step 1. One POST with the batch status is sent when everything is done. See webhooks.

Polling. Call the status endpoint until status is not scraping:

bash
curl --url https://api.webcrawlerapi.com/v2/batch/scrape/6c391693-e566-4b99-97ca-5fa00032e281 \
  --header 'Authorization: Bearer <YOUR_API_KEY>'

completed, failed and total show progress while it runs. Poll every few seconds, not faster.

3. Read the items

Each finished item has a markdown_url (or cleaned_url / html_url, depending on output_formats). Download the file from that link.

Results are paginated. Use limit (up to 500) and pass next as cursor until next is empty:

bash
curl --url 'https://api.webcrawlerapi.com/v2/batch/scrape/<id>?limit=500&cursor=<next>' \
  --header 'Authorization: Bearer <YOUR_API_KEY>'

Handle failures

Some URLs will fail: timeouts, 404s, blocked pages. The batch still finishes. Get only the failed URLs with:

bash
curl --url https://api.webcrawlerapi.com/v2/batch/scrape/<id>/errors \
  --header 'Authorization: Bearer <YOUR_API_KEY>'

Failed URLs are returned with an error code. URLs blocked by robots.txt (when respect_robots_txt is true) are listed under robots_blocked. Resubmit failed URLs in a new batch if you want to retry them.

Control speed and cost

  • max_concurrency limits how many URLs are scraped in parallel. Use it to be gentle on a single website.
  • max_age reuses cached results. Default is 7 days, set 0 to always fetch fresh.
  • DELETE /v2/batch/scrape/:id cancels a running batch. Pages that have not started are not charged.

Pages are billed per page like a crawl. See Pricing.

Next