To use the Webcrawler API you need to obtain an API key. The API key is used to authenticate your requests to the API.

To get an API key sign up and get your key at the API key section (opens in a new tab).

The API key is a secret key that should be kept confidential. Do not share your API key in publicly accessible areas such as GitHub, client-side code, and so on. If you suspect that your API key has been compromised, you can regenerate it at any time.

Webcrawler API uses Bearer Token authentication scheme. You need to include the API key in the Authorization header of your requests, so the header will looks like this:

Authorization: Bearer <PASTE YOUR API KEY HERE>

For example, to make your first request you can use the following curl command:

curl -i --request POST \
  --url https://api.webcrawlerapi.com/v1/crawl \
  --header 'Authorization: Bearer 009b131d7080df28640e' \
  --header 'Content-Type: application/json' \
  --data '{
	"enable_js": true,
	"items_limit": 5,
	"url": "https://example.com",
	"clean_content": false,
}'

If you done everything correctly you will receive a response with a 200 status code and a JSON object with a job_id field. Example:

{
    "job_id": "5f7b1b7b-7b7b-4b7b-8b7b-7b7b7b7b7b7b"
}

Unauthorized error

If you receive an 401 Unauthorized error with body like this:

{
    "error": "Unauthorized"
}

It means that you have provided an invalid API key. Please double-check and make sure that you have copied the key correctly.

If you still have any questions or need help, feel free to contact us at [email protected]. We are always happy to help you!