Batch scrape

Scrape an explicit list of URLs in one async request. The API returns a batch id right away. Poll the status endpoint or set webhook_url to be notified when the batch finishes.

New to batches? Start with the batch scrape guide.

POST /v2/batch/scrape

plain

https://api.webcrawlerapi.com/v2/batch/scrape

Format: JSON
Method: POST

Request

urls - (required) array of URLs to scrape. Maximum is your plan's job items limit (100 during trial).
output_formats - (optional) array of output formats. Valid values: markdown, cleaned, html, links, images. Default is ["markdown"].
main_content_only - (optional) extract only the main content. Default is false.
clean_selectors - (optional) CSS selectors to clean from the output. See clean selectors.
max_concurrency - (optional) max URLs scraped in parallel for this batch. 0 or omitted means no per-batch cap.
ignore_invalid_urls - (optional) skip invalid URLs and report them in invalid_urls. If false, any invalid URL fails the whole request with 400. Default is true.
webhook_url - (optional) public http(s) URL called once when the batch finishes. See webhooks.
max_age - (optional) cache duration in seconds. Default is 604800 (7 days). Set to 0 to bypass the cache. See Caching.
respect_robots_txt - (optional) respect each website's robots.txt. Default is false.

prompt, response_schema and actions are not supported in batch scrape.

URLs are normalized and deduplicated. Duplicates and rejected URLs are returned in invalid_urls.

curl example

bash

curl --request POST \
  --url https://api.webcrawlerapi.com/v2/batch/scrape \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "urls": [
      "https://www.example.com/",
      "https://www.example.com/pricing",
      "https://www.example.com/blog"
    ],
    "output_formats": ["markdown"],
    "max_concurrency": 5,
    "webhook_url": "https://your-server.com/webhook"
  }'

Response

json

{
    "id": "6c391693-e566-4b99-97ca-5fa00032e281",
    "status_url": "https://api.webcrawlerapi.com/v2/batch/scrape/6c391693-e566-4b99-97ca-5fa00032e281",
    "invalid_urls": [
        {
            "url": "not a url",
            "error_code": "invalid_request",
            "error_message": "invalid url"
        }
    ]
}

GET /v2/batch/scrape/:id

plain

https://api.webcrawlerapi.com/v2/batch/scrape/:id

Method: GET

Returns batch progress and one page of items.

Query params:

limit - (optional) items per page, 1 to 500. Default is 100.
cursor - (optional) value of next from the previous page.

bash

curl --request GET \
  --url 'https://api.webcrawlerapi.com/v2/batch/scrape/6c391693-e566-4b99-97ca-5fa00032e281?limit=100' \
  --header 'Authorization: Bearer <YOUR_API_KEY>'

Response

json

{
    "id": "6c391693-e566-4b99-97ca-5fa00032e281",
    "status": "completed",
    "total": 3,
    "completed": 2,
    "failed": 1,
    "canceled": 0,
    "data": [
        {
            "url": "https://www.example.com/",
            "status": "done",
            "page_status_code": 200,
            "title": "Example",
            "markdown_url": "https://data.webcrawlerapi.com/.../page.md"
        },
        {
            "url": "https://www.example.com/blog",
            "status": "error",
            "error_code": "webpage_non_success",
            "error_message": "page returned a non-success status code"
        }
    ],
    "next": ""
}

Fields:

status - batch status: scraping, completed, failed or cancelled.
total, completed, failed, canceled - item counters.
data[].status - item status, for example done or error.
data[].markdown_url, cleaned_url, html_url - links to the output, present for finished items and only for requested formats.
data[].from_cache - true if the result came from cache.
data[].error_code, error_message - set for failed items. See Errors.
next - cursor for the next page. Empty on the last page.

GET /v2/batch/scrape/:id/errors

plain

https://api.webcrawlerapi.com/v2/batch/scrape/:id/errors

Method: GET

Lists URLs that failed. URLs blocked by robots.txt are returned separately in robots_blocked.

json

{
    "errors": [
        {
            "url": "https://www.example.com/blog",
            "error_code": "webpage_non_success",
            "error_message": "page returned a non-success status code"
        }
    ],
    "robots_blocked": []
}

DELETE /v2/batch/scrape/:id

plain

https://api.webcrawlerapi.com/v2/batch/scrape/:id

Method: DELETE

Cancels a running batch. Items that did not start are marked canceled and are not charged.

bash

curl --request DELETE \
  --url https://api.webcrawlerapi.com/v2/batch/scrape/6c391693-e566-4b99-97ca-5fa00032e281 \
  --header 'Authorization: Bearer <YOUR_API_KEY>'

Response:

json

{"status": "canceled"}

When webhook_url is set, the API sends a POST request with the same body as GET /v2/batch/scrape/:id (first 100 items) once the batch finishes. For larger batches, fetch the remaining pages with cursor.

Error Responses

400 Bad Request - invalid parameters, no valid URLs, or batch already finished (on cancel)
401 Unauthorized - invalid or missing API key
402 Payment Required - insufficient account balance
403 Forbidden - organization suspended
404 Not Found - batch does not exist
500 Internal Server Error - server-side error