Scrape a page

Endpoint to scrape a single webpage.

https://api.webcrawlerapi.com/v2/scrape

Format: JSON
Method: POST

Request#

Available request params:

• url - (required) The URL of the webpage to scrape.
• prompt - (optional) A prompt to run on the scraped content. This can be used to extract specific information or to format the output. Charged per token consumed, depending on page content — see Pricing.
• output_formats - (optional) array of output formats to generate. Valid values: markdown, cleaned, html, links.
• main_content_only - (optional) Extract only the main content of article or blog post. When set to true, the scraper will focus on extracting the primary article content while filtering out navigation, sidebars, ads, and other non-essential elements. Default is false.
• clean_selectors - (optional) CSS selectors to clean from the output. Read more about advanced cleaning in clean selectors.
• respect_robots_txt - (optional) if set to true, the scraper will respect the website's robots.txt file and return an error if the URL is disallowed. Default is false.

Query params:

• async - (optional) if set to true (/v2/scrape?async=true), the request returns a job id immediately instead of waiting for the result. Fetch the result with GET /v2/scrape/{id}. See Async Requests.

Example:

{
    "url": "https://www.example.com",
    "output_formats": ["markdown", "html", "links"],
    "main_content_only": true,
    "clean_selectors": ".advertisement,.footer",
    "respect_robots_txt": true
}

curl example#

curl --request POST \
  --url https://api.webcrawlerapi.com/v2/scrape \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://www.example.com",
    "output_formats": ["markdown", "html", "links"],
    "main_content_only": true,
    "clean_selectors": ".advertisement,.footer",
    "respect_robots_txt": true
  }'

Response#

The response will contain a status and all requested outputs in the same object.

{
    "success": true,
    "status": "done",
    "markdown": "## Example Product\n\nThis is an example product page. It has a title, a price, and a description.",
    "raw_content": "<html><body><h1>Example Product</h1></body></html>",
    "links": [
        "https://www.example.com/pricing",
        "https://www.example.com/contact"
    ],
    "page_status_code": 200,
    "page_title": "Example Product"
}

Possible response fields:

• markdown when markdown was requested
• cleaned_content when cleaned was requested
• raw_content when html was requested
• links when links was requested
• structured_data when prompt is used

Scrape errors#

If the scrape fails, the response will have 200 status code but the success will be false, the error_code and error_message will be set.

If the domain name cannot be resolved, the API returns 400 Bad Request with the name_not_resolved error code:

{
    "success": false,
    "error_code": "name_not_resolved",
    "error_message": "unable to resolve domain name"
}

Read more about error codes in Error section.

Error Responses#

• 400 Bad Request - Invalid parameters or missing required fields
• 401 Unauthorized - Invalid or missing API key
• 402 Payment Required - Insufficient account balance
• 500 Internal Server Error - Server-side error

Get scrape result#

GET https://api.webcrawlerapi.com/v2/scrape/{id}

Returns the result of a scrape started with async=true. While the job is running the response is {"status": "pending"}. Once finished, it has the same fields as the synchronous response above.

Refer to Async Requests for more information about handling asynchronous scraping jobs.

Need LLM-cleaned markdown (main content only, no boilerplate) instead of raw markdown? See GET/POST /markdown.