Turn any website into structured data - with a single prompt

Describe what you want. Our AI agent browses the site, follows the links, and returns clean JSON. No selectors. No maintenance.

https://

No credit card required · First run free

Webcrawler Agent Use Cases

One prompt. Dozens of pages crawled. Structured JSON out.

Request

$ curl -X POST https://api.webcrawlerapi.com/v1/agent \
  -H "Authorization: Bearer <API_TOKEN>" \
  -d '{
    "prompt": "Find all customers listed on the website",
    "urls": ["https://mintlify.com"],
    "max_spend_usd": 0.5
  }'

Response

{
  "customers": [

Get Your API Key

How Webcrawler Agent works

Works like a researcher. Returns like an API.

Write a prompt

Describe what you want in plain English. Add a seed URL and optional JSON schema to enforce the output shape.

Agent browses the web

It searches the web, navigates links, reads pages, and makes decisions about what information is relevant - like a researcher working for you.

Get structured JSON

Results arrive as clean, schema-validated JSON. Ready to use in your AI agent, n8n workflow, app, or vector store. No cleanup needed.

Developer First

Integrate in 60 seconds

One API call. Send a prompt, get structured data back.

agent_quickstart

curl -X POST https://api.webcrawlerapi.com/v1/agent \
  -H "Authorization: Bearer <API_TOKEN>" \
  -d '{
    "prompt": "Find all customers listed on the website",
    "urls": ["https://mintlify.com"],
    "max_spend_usd": 0.5
  }'

Get Your API Key Read the docs

Built for production

Powerful controls out of the box

Smart defaults so you get results immediately, with fine-grained control when you need it.

Autonomous link following

Agent decides which pages to visit based on your prompt. Set seed_urls_only: true to lock it to the URLs you provide.

{ "seed_urls_only": false, "max_pages": 50 }

Output schema enforcement

Pass a JSON Schema and get back exactly the shape you defined - no post-processing, no guessing.

Spend control

Set max_spend_usd per run. Agent stops at your budget. No surprise bills.

Your choice of LLM

Gemini, GPT, or Claude. Same API - swap the model to tune cost vs accuracy.

By the numbers

Predictable cost. Reliable results.

Pay only for what the agent actually uses. Every run tracks spend in real time and stops at your budget cap.

Affordable

$0.03

Average cost per run

Fast

29s

Median time to result

Reliable

98.2%

Successful runs

Structured

JSON

Schema-validated output, always

Pricing

Simple, transparent pricing

Pick a plan that fits your volume. All plans include full access to Webcrawler Agent.

Starter

Great for small projects

$29/month

Scrape $0.002 / page
Unlimited proxy included
Up to 10 parallel requests
Pay only for successful requests
Content cleaning included
Webcrawler agent

Try for free

Professional

Best for growing teams

Save 25%

$99/month

Scrape $0.0015 / page
Unlimited proxy included
Up to 20 parallel requests
Pay only for successful requests
Content cleaning included
Webcrawler agent

Try for free

Business

For high-volume crawling

Save 50%

$499/month

Scrape $0.001 / page
Unlimited proxy included
Up to 50 parallel requests
Pay only for successful requests
Content cleaning included
Webcrawler agent

Try for free

* Webcrawler Agent is available exclusively on monthly subscription plans.

Need a custom quota or special pricing? Contact us and we'll set up a plan that fits your needs.

Frequently asked questions

Everything you need to know before your first agent run

How is this different from regular web scraping?

Standard scraping fetches a specific URL with a specific selector. The agent acts like a researcher: it reads pages, decides which links are relevant based on your prompt, follows them autonomously, and synthesizes the results into structured JSON - all without you writing a single selector.

How much does an agent run cost?

You set the budget. The agent tracks spend in real time and stops when it hits your max_spend_usd cap. Most runs cost well under $0.10. You only pay for successful page fetches that the agent actually uses.

What happens if the agent hits the spending limit?

It stops gracefully and returns whatever data it has collected so far. You'll see status: "done" and the partial result in the data field. Nothing is billed beyond your cap.

Can I force the agent to stay on a single page?

Yes - set seed_urls_only: true and the agent will only process the URLs you provided, without following any links. Useful for extracting structured data from a known page without autonomous browsing.

Which LLM models are available?

Gemini 3.1 Flash, GPT-5.4 mini, GPT-5.4, GPT-5.5, and Claude Sonnet 4.6. Pass the model ID in the model field. See the full model list in the docs.

How do I guarantee the output shape?

Pass an output_schema JSON Schema in your request. The agent will fill in your schema rather than inventing its own shape. This guarantees the data field matches your expected structure exactly.

Start extracting in 60 seconds

No setup. No selectors. Write a prompt and go.

Get API Key Read the docs