Looking for an overview with examples? See the JavaScript web crawler and JavaScript web scraper pages.
Installation
npm i webcrawlerapi-jsUsage
Synchronous Crawling
The synchronous method waits for the crawl to complete and returns all data at once.
import webcrawlerapi from "webcrawlerapi-js";
const client = new webcrawlerapi.WebcrawlerClient("YOUR_API_KEY");
// Synchronous crawling
const result = await client.crawl({
"url": "https://stripe.com/",
"output_formats": ["markdown"],
"items_limit": 10
});
for (const item of result.job_items) {
item.getContent().then((content) => {
console.log(content.slice(0, 100));
})
}
console.log(result);Asynchronous Crawling
The asynchronous method returns a job ID immediately and allows you to check the status later.
import webcrawlerapi from "webcrawlerapi-js";
const client = new webcrawlerapi.WebcrawlerClient("YOUR_API_KEY");
// Start the async crawl job
const job = await client.crawlAsync({
"url": "https://stripe.com/",
"output_formats": ["markdown"],
"items_limit": 10
});
// Get the job ID
const jobId = job.id;
// Check job status
let jobStatus = await client.getJob(jobId);
console.log(jobStatus);
// You can poll the job status until it's complete
while (jobStatus.status === 'in_progress') {
await new Promise(resolve => setTimeout(resolve, jobStatus.recommended_pull_delay_ms));
jobStatus = await client.getJob(jobId);
}
console.log('Final result:', jobStatus);Options
Both methods support the following options:
url: The target URL to crawloutput_formats: Array of output formats to generate (e.g.['markdown'],['html'],['markdown', 'links'])items_limit: Maximum number of pages to crawlwhitelist_regexp: Regular expression for allowed URLsblacklist_regexp: Regular expression for blocked URLswebhook_url: URL to receive notifications when the job completes
Batch scrape
Scrape a list of URLs in one async request. Pass an array as url to scrape() to start a batch and wait for it to finish.
const batch = await client.scrape({
"url": ["https://example.com/", "https://example.com/pricing"],
"output_formats": ["markdown"],
"max_concurrency": 5
});
console.log(batch.status, batch.completed, batch.failed);scrape({ url: [...] }): start a batch and poll until it finishesscrapeBatchAsync({ urls }): start a batch and return itsidimmediatelygetBatch(id, { limit, cursor }): get progress and one page of itemsgetBatchErrors(id): list failed URLscancelBatch(id): cancel a running batch (pages that did not start are not charged)
See the batch scrape guide and the batch scrape API.
GetContent
The job item contains a link to its content. For convenience, there is a getContent() method that allows you to easily access this content. Here's an example:
const result = await client.crawl({
"url": "https://stripe.com/",
"output_formats": ["markdown"],
"items_limit": 10
});
for (const item of result.job_items) {
item.getContent().then((content) => {
console.log(content.slice(0, 100));
})
}This method retrieves the full content associated with the job, which can be useful for processing or displaying the job's data.