Pages are not just HTML
Client-rendered content, lazy loading and cookie walls mean a plain fetch returns an empty shell.
Crawl API
Crawl API: turn a whole website into clean markdown with one request
Point the crawl API at your docs or help center. A 2,000-page site costs about $3, and you only pay for pages that succeed. Every page comes back as markdown, ready for your knowledge base or support bot.
https://
- ✓ No credit card to try
- ✓ Failed pages are free
- ✓ Webhook or polling
One request, whole site
0 / 10 pages
seed URL → follows links → every page back as markdown
job 23b81e21 · running
Trusted by 1,000+ developers building AI support bots and knowledge products
The problem
Why a hand-rolled crawler wastes your month
Sites block bots
Rate limits, blocks and flaky pages mean retries and proxies. That work never ends.
Noise ruins retrieval
Menus and footers in your chunks make your bot cite the wrong page. Cleaning is a project by itself.
Weighing it up? Read should you build a crawler or use a crawling API.
The output
Pages in, clean markdown out
Menus, banners and footers end up in your embeddings if you do not remove them. The crawl API gives you the page as markdown, ready to chunk and embed.
The page: docs.acme.com/api/authentication
Cookie banner: "We use cookies…"
Top navigation: Product · Pricing · Docs · Login
Sidebar menu (40 links)
Authentication
Every request needs an API key in the Authorization header. Create a key in the dashboard…
The part you want
Newsletter signup · Related posts · Footer (60 links)
What the crawl API returns (markdown)
# Authentication
Every request needs an API key in the
Authorization header.
## Create a key
1. Open **Dashboard → Access keys**
2. Click **New key**
3. Copy it. It is shown once.
```bash
curl -H "Authorization: Bearer $KEY" \
https://api.acme.com/v1/me
```How it works
How the crawl API works: one request, every page
1. Send a seed URL
POST /v1/crawlwithurlanditems_limit. You get a jobidback right away.2. The crawler follows links
It discovers pages from the seed, staying inside yourmax_depth,whitelist_regexp,blacklist_regexpanditems_limit.3. Get results by webhook or polling
Setwebhook_urlor pollGET /v1/job/{id}. Each job item has its URL, status and a link to each format you asked for. Read about async requests.
Control what gets crawled
Change the settings and watch the request update.
max_depth
output_formats
Worst case for this job on Professional: 100 × $0.0015 = $0.15. You only pay for pages that succeed.
Your request
curl --request POST \
--url https://api.webcrawlerapi.com/v1/crawl \
--header 'Authorization: Bearer <YOUR_API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
"url": "https://docs.acme.com/",
"items_limit": 100,
"output_formats": ["markdown"],
"max_depth": 2
}'| Param | What it does |
|---|---|
| url | Seed URL where the crawler starts. |
| items_limit | Stops the crawl when it reaches this number of pages. |
| max_depth | How many links deep to follow. 0 means the seed page only. |
| whitelist_regexp | Only URLs matching this pattern are crawled. |
| blacklist_regexp | URLs matching this pattern are skipped. |
| respect_robots_txt | Skip pages disallowed by robots.txt. Default false. |
| output_formats | Any of markdown, html, links. Default markdown. |
| webhook_url | Receive a POST when the job finishes. |
Fresh or cached
Results are cached for 7 days by default. Set max_age to 0 to force a fresh crawl. Details in the caching docs.
Code
Crawl a website in a few lines of code
curl --request POST \
--url https://api.webcrawlerapi.com/v1/crawl \
--header 'Authorization: Bearer <YOUR_API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
"url": "https://stripe.com/",
"webhook_url": "https://yourserver.com/webhook",
"items_limit": 10,
"output_formats": ["markdown"],
"max_depth": 2
}'You get a job id back
{
"id": "23b81e21-c672-4402-a886-303f18de9555"
}GET /v1/job/{id} when it is done
{
"id": "23b81e21-c672-4402-a886-303f18de9555",
"status": "done",
"url": "https://docs.acme.com/",
"items_limit": 10,
"output_formats": ["markdown"],
"job_items": [
{
"original_url": "https://docs.acme.com/api/authentication",
"page_status_code": 200,
"status": "done",
"title": "Authentication | Acme Docs",
"markdown_content_url": "https://data.webcrawlerapi.com/markdown/..."
}
]
}Also available in PHP and Java SDKs, plus n8n, Make, Zapier, LangChain and MCP integrations. See the JavaScript and Python SDK docs, or the guides for a JavaScript web crawler and a Python web crawler.
Formats
Output formats for every crawled page
markdown
Default. Headings, links and lists are kept. Best for LLM context and knowledge-base chunks.
html
Raw HTML, untouched.
links
The links found on each page, for building your own sitemap or URL list.
Use cases
What teams build with the crawl API
Support bot knowledge base
Before: the bot answers from memory and gets your product wrong.
After: crawl your docs to markdown, then chunk and embed it.
whitelist_regexp: "/docs/"Help-center sync
Before: articles change and the bot keeps quoting the old ones.
After: re-crawl on a schedule, or use Feeds to get only changed pages.
max_age: 0Docs ingestion for AI assistants
Before: an unbounded crawl runs up the bill.
After: cap cost and scope with a page limit and a depth limit.
items_limit + max_depthSite inventory and link maps
Before: no clear list of what is on the site.
After: get every link on every page for a sitemap or URL audit.
output_formats: ["links"]Learn more: crawl a documentation site for a knowledge base, keep it in sync, crawling vs scraping.
By the numbers
Reliable infrastructure, proven in production.
Tracking usage, crawl success rates, average response times, uptime percentages, quality scores, cost metrics, API performance, reliability data, and live trends for all your requests.
Active builders
100+
Teams building in production
Extraction quality
98%
Success rate
Fast turnaround
2.5s
Average crawling time
Platform stability
99.98%
Uptime
Which one
Crawl API vs scraping API vs DIY
| Crawl API | Scraping API | DIY Playwright | |
|---|---|---|---|
| Starts from | One seed URL | A URL you list | Whatever you code |
| Finds new pages | Yes, by following links | No | You write the link logic |
| Best for | Whole sites, docs, help centers | Known pages, product pages | Custom flows and logins |
| Retries and proxies | Handled for you | Handled for you | You maintain them |
| Cost model | Per successful page | Per request | Servers and your time |
Comparing providers? Read Firecrawl alternatives and all crawler APIs compared.
Built for sites that fight back
Handled automatically
Residential proxies
Automatic retries
Rate limit handling
Real headless browsers
JavaScript rendering
CAPTCHA solving
Anti-bot bypass
Fastest path selection
Skip browser maintenance. Per-site concurrency is shared and capped, so your crawl will not hammer the target site. See rate limits.
Pricing
Simple, transparent pricing
Start with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.
Pay as you go
No monthly commitment — pay only for what you use
$0/month
- From $0.002 / page
- LLM-cleaned markdown: $0.006 / page
- Unlimited proxy included
- Up to 5 parallel requests
- Pay only for successful requests
- Webcrawler agent
Starter
Great for small projects
$29/month
- From $0.002 / page
- LLM-cleaned markdown: $0.006 / page
- Unlimited proxy included
- Up to 10 parallel requests
- Pay only for successful requests
- Webcrawler agent
Professional
Best for growing teams
$99/month
- From $0.0015 / page
- LLM-cleaned markdown: $0.0055 / page
- Unlimited proxy included
- Up to 20 parallel requests
- Pay only for successful requests
- Webcrawler agent
Business
For high-volume crawling
$499/month
- From $0.001 / page
- LLM-cleaned markdown: $0.005 / page
- Unlimited proxy included
- Up to 50 parallel requests
- Pay only for successful requests
- Webcrawler agent
Need a custom quota or special pricing? Contact us and we'll set up a plan that fits your needs.
Crawl API FAQ
Common questions about crawling websites with an API
What is a crawl API?
A crawl API starts at a seed URL, follows links across the site, and returns the content of each page it finds. With WebCrawlerAPI that's one POST /v1/crawl request. You get markdown, HTML or links for every page.
What's the difference between a crawl API and a scraping API?
A scraping API fetches the one URL you name (our /v2/scrape). A crawl API discovers pages on its own from a seed URL and processes many of them (our /v1/crawl). Use scrape for known URLs and crawl for whole sites.
How do I crawl an entire website with an API?
Send the homepage or docs root as url, set items_limit high enough for the site, optionally max_depth, and a webhook_url. The job runs asynchronously and posts the result to your webhook when it finishes.
How do I limit which pages get crawled?
Use whitelist_regexp and blacklist_regexp to include or skip URL patterns, max_depth to limit link depth, items_limit to cap the page count, and respect_robots_txt to follow the site robots.txt rules.
Is the crawl synchronous or asynchronous?
Asynchronous. The API returns a job id right away, then you get a webhook or poll GET /v1/job/{id}. The JavaScript and Python SDK crawl() methods wait for you; crawlAsync and crawl_async return immediately.
How much does crawling cost?
You pay per successfully crawled page: $0.002 on Starter, $0.0015 on Professional and $0.001 on Business. Failed pages are not charged. Some sites are priced separately, see the pricing docs for details.
Can I use crawled pages in a knowledge base or support bot?
Yes. Markdown keeps headings and lists, which helps chunking. Crawl your docs or help center, then chunk and embed the markdown.
Does the crawler respect robots.txt?
Only if you set respect_robots_txt to true. The default is false. Per-site concurrency is shared and capped, so a crawl will not hammer the target website.
How do I keep crawled content up to date?
Re-run the crawl job; set max_age to 0 to skip the cache and get fresh content. To detect only the pages that changed, use Feeds, which deliver updates by webhook, RSS or JSON.
Can I try the crawl API before paying?
Yes. Sign up in the dashboard and create an account. No credit card required.
Crawl your first site in five minutes
Get an API key, paste the curl above, and get markdown for every page of your docs. Or try it in the playground.