WebCrawlerAPI

SDKs/Official SDKs/JavaScript / TypeScript

JavaScript and TypeScript (Node.js)

Learn how to use the WebCrawler API JavaScript SDK to crawl websites and extract data.

Looking for an overview with examples? See the JavaScript web crawler and JavaScript web scraper pages.

Installation

plain
npm i webcrawlerapi-js

Usage

Synchronous Crawling

The synchronous method waits for the crawl to complete and returns all data at once.

javascript
import webcrawlerapi from "webcrawlerapi-js";

const client = new webcrawlerapi.WebcrawlerClient("YOUR_API_KEY");

// Synchronous crawling
const result = await client.crawl({
    "url": "https://stripe.com/",
    "output_formats": ["markdown"],
    "items_limit": 10
});

for (const item of result.job_items) {
    item.getContent().then((content) => {
        console.log(content.slice(0, 100));
    })
}
console.log(result);

Asynchronous Crawling

The asynchronous method returns a job ID immediately and allows you to check the status later.

javascript
import webcrawlerapi from "webcrawlerapi-js";

const client = new webcrawlerapi.WebcrawlerClient("YOUR_API_KEY");

// Start the async crawl job
const job = await client.crawlAsync({
    "url": "https://stripe.com/",
    "output_formats": ["markdown"],
    "items_limit": 10
});

// Get the job ID
const jobId = job.id;

// Check job status
let jobStatus = await client.getJob(jobId);
console.log(jobStatus);

// You can poll the job status until it's complete
while (jobStatus.status === 'in_progress') {
    await new Promise(resolve => setTimeout(resolve, jobStatus.recommended_pull_delay_ms));
    jobStatus = await client.getJob(jobId);
}

console.log('Final result:', jobStatus);

Options

Both methods support the following options:

  • url: The target URL to crawl
  • output_formats: Array of output formats to generate (e.g. ['markdown'], ['html'], ['markdown', 'links'])
  • items_limit: Maximum number of pages to crawl
  • whitelist_regexp: Regular expression for allowed URLs
  • blacklist_regexp: Regular expression for blocked URLs
  • webhook_url: URL to receive notifications when the job completes

Batch scrape

Scrape a list of URLs in one async request. Pass an array as url to scrape() to start a batch and wait for it to finish.

javascript
const batch = await client.scrape({
    "url": ["https://example.com/", "https://example.com/pricing"],
    "output_formats": ["markdown"],
    "max_concurrency": 5
});
console.log(batch.status, batch.completed, batch.failed);
  • scrape({ url: [...] }): start a batch and poll until it finishes
  • scrapeBatchAsync({ urls }): start a batch and return its id immediately
  • getBatch(id, { limit, cursor }): get progress and one page of items
  • getBatchErrors(id): list failed URLs
  • cancelBatch(id): cancel a running batch (pages that did not start are not charged)

See the batch scrape guide and the batch scrape API.

GetContent

The job item contains a link to its content. For convenience, there is a getContent() method that allows you to easily access this content. Here's an example:

javascript
const result = await client.crawl({
    "url": "https://stripe.com/",
    "output_formats": ["markdown"],
    "items_limit": 10
});

for (const item of result.job_items) {
    item.getContent().then((content) => {
        console.log(content.slice(0, 100));
    })
}

This method retrieves the full content associated with the job, which can be useful for processing or displaying the job's data.