---
title: AI Agent Integration Guide
description: A complete guide for AI coding agents to integrate WebCrawlerAPI into an existing codebase.
---

This guide is written for AI coding agents (Cursor, Claude Code, Copilot, etc.) to integrate WebCrawlerAPI into an existing project. Follow the steps below to add web scraping and crawling capabilities to a codebase.

---

## Step 1 — Obtain an API Key

**Before writing any code**, the user must have an API key.

Instruct the user:
> Please go to [dash.webcrawlerapi.com/access](https://dash.webcrawlerapi.com/access), sign in or create a free account, and copy your API key.

Store the key as an environment variable — never hardcode it:

```bash
WEBCRAWLERAPI_KEY=your_key_here
```

All requests use Bearer token authentication:

```
Authorization: Bearer <API_KEY>
```

---

## Step 2 — Choose the Right Endpoint

WebCrawlerAPI has two primary endpoints:

| Use case | Endpoint | Pattern |
|---|---|---|
| Scrape a single URL | `POST /v2/scrape` | Synchronous (instant result) |
| Crawl an entire site | `POST /v1/crawl` | Async (returns job ID) |

---

## Step 3 — Check for an Existing SDK

Before making raw HTTP calls, check if there is an SDK for the project's language. SDKs handle polling, retries, and content fetching automatically.

| Language | Package | Docs |
|---|---|---|
| JavaScript / TypeScript | `npm i webcrawlerapi-js` | [JS/TS SDK](/docs/sdk/js) |
| Python | `pip install webcrawlerapi` | [Python SDK](/docs/sdk/python) |
| PHP | `composer require webcrawlerapi/sdk` | [PHP SDK](/docs/sdk/php) |
| .NET / C# | `dotnet add package WebCrawlerApi` | [.NET SDK](/docs/sdk/dotnet) |

If the project uses one of those languages, **use the SDK** — skip to the SDK examples in [Step 5](#step-5--sdk-examples). For all other languages, use the raw HTTP API described in [Step 4](#step-4--raw-http-api).

---

## Step 4 — Raw HTTP API

### 4.1 — Scrape a Single Page (`POST /v2/scrape`)

Use this to extract content from one URL immediately.

**Endpoint:**
```
POST https://api.webcrawlerapi.com/v2/scrape
```

**Request body:**

```json
{
  "url": "https://example.com/page",
  "output_formats": ["markdown"],
  "main_content_only": true
}
```

| Parameter | Type | Required | Description |
|---|---|---|---|
| `url` | string | yes | URL to scrape |
| `output_formats` | array | no | `["markdown"]`, `["html"]`, `["cleaned"]`, `["links"]` or any combination. Default: `["markdown"]` |
| `prompt` | string | no | Extract specific information via AI prompt (+$0.002/request) |
| `main_content_only` | boolean | no | Strip nav, ads, sidebars — keep article body only |
| `clean_selectors` | string | no | CSS selectors to remove (e.g. `.ads,.footer`) |
| `respect_robots_txt` | boolean | no | Honour robots.txt rules. Default: `false` |

**Example curl:**

```bash
curl --request POST \
  --url https://api.webcrawlerapi.com/v2/scrape \
  --header 'Authorization: Bearer $WEBCRAWLERAPI_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://example.com/page",
    "output_formats": ["markdown"],
    "main_content_only": true
  }'
```

**Successful response:**

```json
{
  "success": true,
  "status": "done",
  "markdown": "## Page Title\n\nContent here...",
  "page_status_code": 200,
  "page_title": "Page Title"
}
```

**On failure** (`success: false`):

```json
{
  "success": false,
  "error_code": "name_not_resolved",
  "error_message": "Unable to resolve domain name"
}
```

Always check `success` before consuming content. See [Errors](/docs/errors) for all error codes.

**Async scraping:** Add `?async=true` to the URL to get a job ID immediately, then poll:

```bash
# Start
curl -X POST 'https://api.webcrawlerapi.com/v2/scrape?async=true' ...
# Returns: { "id": "<JOB_ID>" }

# Poll
curl https://api.webcrawlerapi.com/v2/scrape/<JOB_ID> \
  --header 'Authorization: Bearer $WEBCRAWLERAPI_KEY'
```

---

### 4.2 — Crawl a Website (`POST /v1/crawl` + `GET /v1/job/:id`)

Use this to crawl multiple pages of a site. This is always asynchronous.

**Step A — Start the crawl:**

```
POST https://api.webcrawlerapi.com/v1/crawl
```

**Request body:**

```json
{
  "url": "https://docs.example.com/",
  "items_limit": 50,
  "output_formats": ["markdown"],
  "main_content_only": true,
  "whitelist_regexp": ".*docs\\.example\\.com.*",
  "max_depth": 3,
  "webhook_url": "https://yourserver.com/webhook"
}
```

| Parameter | Type | Required | Description |
|---|---|---|---|
| `url` | string | yes | Seed URL where crawling starts |
| `items_limit` | integer | yes | Max pages to crawl |
| `output_formats` | array | no | Same values as `/v2/scrape` |
| `main_content_only` | boolean | no | Extract article body only |
| `whitelist_regexp` | string | no | Only crawl URLs matching this regex |
| `blacklist_regexp` | string | no | Skip URLs matching this regex |
| `max_depth` | integer | no | Crawl depth limit (0 = seed page only) |
| `respect_robots_txt` | boolean | no | Honour robots.txt. Default: `false` |
| `webhook_url` | string | no | POST callback when job completes |

**Response:**

```json
{ "id": "23b81e21-c672-4402-a886-303f18de9555" }
```

Save this `id` — it is the job ID used to check results.

---

**Step B — Poll job status:**

```
GET https://api.webcrawlerapi.com/v1/job/:id
```

```bash
curl https://api.webcrawlerapi.com/v1/job/23b81e21-c672-4402-a886-303f18de9555 \
  --header 'Authorization: Bearer $WEBCRAWLERAPI_KEY'
```

**Job statuses:** `new` → `in_progress` → `done` | `error`

Poll until `status` is `done` or `error`. Use the `recommended_pull_delay_ms` field from the response to space out polls.

**Response when done:**

```json
{
  "id": "23b81e21-c672-4402-a886-303f18de9555",
  "status": "done",
  "url": "https://docs.example.com/",
  "items_limit": 50,
  "output_formats": ["markdown"],
  "created_at": "2024-12-15T10:26:13.893Z",
  "finished_at": "2024-12-15T10:26:37.118Z",
  "job_items": [
    {
      "id": "a46f3117-f97a-4ca2-a434-6cfdcd022b72",
      "status": "done",
      "original_url": "https://docs.example.com/getting-started",
      "page_status_code": 200,
      "markdown_content_url": "https://data.webcrawlerapi.com/markdown/...",
      "title": "Getting Started",
      "cost": 2000,
      "created_at": "2024-12-15T10:26:17.941Z"
    }
  ]
}
```

**Fetch content from a job item:**

Each item contains a `markdown_content_url` (or `raw_content_url` / `cleaned_content_url`). Fetch those URLs with the same `Authorization` header to get the actual page content.

```bash
curl "<markdown_content_url>" \
  --header 'Authorization: Bearer $WEBCRAWLERAPI_KEY'
```

See [Job reference](/docs/api/job) and [Async Requests](/docs/async-requests) for full details.

---

## Step 5 — SDK Examples

### JavaScript / TypeScript

```typescript
import webcrawlerapi from "webcrawlerapi-js";

const client = new webcrawlerapi.WebcrawlerClient(process.env.WEBCRAWLERAPI_KEY!);

// Single page scrape (use fetch or the crawl API — JS SDK uses crawl)
const job = await client.crawl({
  url: "https://docs.example.com/",
  output_formats: ["markdown"],
  items_limit: 20,
});

for (const item of job.job_items) {
  const content = await item.getContent();
  console.log(content?.slice(0, 200));
}
```

Full docs: [JS/TS SDK](/docs/sdk/js)

---

### Python

```python
import os
from webcrawlerapi import WebCrawlerAPI

crawler = WebCrawlerAPI(api_key=os.environ["WEBCRAWLERAPI_KEY"])

# Synchronous — waits for completion
job = crawler.crawl(
    url="https://docs.example.com/",
    output_formats=["markdown"],
    items_limit=20,
)

for item in job.job_items:
    print(item.title, item.original_url)
    # Fetch actual content:
    # requests.get(item.markdown_content_url, headers={"Authorization": ...})
```

Full docs: [Python SDK](/docs/sdk/python)

---

### PHP

```php
use WebCrawlerAPI\WebCrawlerAPI;

$crawler = new WebCrawlerAPI($_ENV['WEBCRAWLERAPI_KEY']);

$job = $crawler->crawl(
    url: 'https://docs.example.com/',
    scrapeType: 'markdown',
    itemsLimit: 20
);

foreach ($job->jobItems as $item) {
    $content = $item->getContent();
    echo $item->title . "\n";
}
```

Full docs: [PHP SDK](/docs/sdk/php)

---

### .NET / C#

```csharp
using WebCrawlerApi;

var crawler = new WebCrawlerApiClient(Environment.GetEnvironmentVariable("WEBCRAWLERAPI_KEY")!);

var job = await crawler.CrawlAndWaitAsync(
    url: "https://docs.example.com/",
    scrapeType: "markdown",
    itemsLimit: 20
);

foreach (var item in job.JobItems)
{
    var content = await item.GetContentAsync();
    Console.WriteLine(item.Title);
}
```

Full docs: [.NET SDK](/docs/sdk/dotnet)

---

## Step 6 — Integration Patterns

### Pattern A: Single-page content extraction

Best for: fetching one URL on demand (e.g. user pastes a URL, agent looks up a page).

Use `POST /v2/scrape` with `output_formats: ["markdown"]` and `main_content_only: true`. The response is synchronous — no polling needed.

### Pattern B: Build a knowledge base from a documentation site

Best for: ingesting an entire docs site into a vector store or RAG pipeline.

1. `POST /v1/crawl` with `whitelist_regexp` scoped to the docs subdomain and a reasonable `items_limit`.
2. Poll `GET /v1/job/:id` until `status === "done"`.
3. Iterate `job_items`, fetch each `markdown_content_url`, chunk and embed.

### Pattern C: Webhook-driven pipeline

Best for: background jobs where the agent does not wait.

Pass `webhook_url` in the crawl request. When the job finishes, the server POSTs the full job object to that URL. Handle it in a route, then process `job_items`.

---

## Step 7 — Error Handling Checklist

When integrating, make sure to handle these cases:

- **401 Unauthorized** — API key missing or invalid. Prompt the user to check [dash.webcrawlerapi.com/access](https://dash.webcrawlerapi.com/access).
- **402 Payment Required** — Account balance is zero. Direct the user to add credits at [dash.webcrawlerapi.com](https://dash.webcrawlerapi.com).
- **`success: false`** on `/v2/scrape` — The page itself failed (blocked, DNS error, etc.). Check `error_code`.
- **Job item `status: "error"`** — Individual pages may fail inside a crawl job. Log `last_error` and continue processing successful items.
- **Polling timeout** — Set a maximum number of poll attempts. Use `recommended_pull_delay_ms` from the job response as the polling interval.

Full error code reference: [Errors](/docs/errors)

---

## API Reference Links

| Resource | URL |
|---|---|
| `POST /v2/scrape` | [/docs/api/scrape](/docs/api/scrape) |
| `POST /v1/crawl` | [/docs/api/crawl](/docs/api/crawl) |
| `GET /v1/job/:id` | [/docs/api/job](/docs/api/job) |
| Async Requests & Webhooks | [/docs/async-requests](/docs/async-requests) |
| Output formats | [/docs/crawling-types](/docs/crawling-types) |
| Access key setup | [/docs/access-key](/docs/access-key) |
| Error codes | [/docs/errors](/docs/errors) |
| Rate limits | [/docs/rate-limits](/docs/rate-limits) |