Python SDK
Python web crawler: crawl any website in 5 lines of code
pip install webcrawlerapi and get every page of a site as clean markdown. No headless browsers, no Scrapy spiders to maintain. You only pay for pages that succeed.
- ✓ Python 3.6+
- ✓ Only dependency: requests
- ✓ No credit card to try
- ✓ Failed pages are free
crawler.py
from webcrawlerapi import WebCrawlerAPI
client = WebCrawlerAPI(api_key="YOUR_API_KEY")
job = client.crawl(
url="https://docs.acme.com/",
output_formats=["markdown"],
items_limit=50,
max_depth=2,
)
print(f"{job.status}: {len(job.job_items)} pages")
for item in job.job_items:
if item.status == "done":
print(item.original_url)
print(item.content[:200])One request, whole site
0 / 10 pages
seed URL → follows links → every page back as markdown
Quickstart
How to crawl a website with Python
1. Install the SDK
pip install webcrawlerapi2. Get an API key
Sign up and copy the key from the dashboard. No credit card needed to try it.3. Call crawl()
Pass a seed URL anditems_limit. Loop overjob.job_itemsand readitem.content.
The problem
Why a DIY Python crawler wastes your month
The crawler is the easy 10%
Parsed HTML is not LLM-ready
Prefer to build it yourself first? Read how to crawl a website with Python or should you build a crawler or use a crawling API.
The output
Pages in, clean markdown out
item.content is the page as markdown. No BeautifulSoup selectors, no boilerplate in your embeddings.
The page: docs.acme.com/api/authentication
Authentication
Every request needs an API key in the Authorization header. Create a key in the dashboard…
The part you want
What the crawl API returns (markdown)
# Authentication
Every request needs an API key in the
Authorization header.
## Create a key
1. Open **Dashboard → Access keys**
2. Click **New key**
3. Copy it. It is shown once.
```bash
curl -H "Authorization: Bearer $KEY" \
https://api.acme.com/v1/me
```Code
Python examples you can paste and run
from webcrawlerapi import WebCrawlerAPI
client = WebCrawlerAPI(api_key="YOUR_API_KEY")
job = client.crawl(
url="https://docs.acme.com/",
output_formats=["markdown"],
items_limit=50,
max_depth=2,
)
print(f"{job.status}: {len(job.job_items)} pages")
for item in job.job_items:
if item.status == "done":
print(item.original_url)
print(item.content[:200])What job and item give you
job.id # "23b81e21-c672-..."
job.status # new | in_progress | done | error | cancelled
job.job_items # list of JobItem
job.recommended_pull_delay_ms # polling interval hint
item = job.job_items[0]
item.original_url # "https://docs.acme.com/api/authentication"
item.title # "Authentication | Acme Docs"
item.page_status_code # 200
item.status # "done"
item.content # page content, best available format
item.get_markdown() # fetch markdown
item.get_html() # fetch raw HTMLFull reference in the Python SDK docs, or read the source on GitHub. Prefer plain HTTP? See the crawl API page for curl.
Formats
Output formats for every page
markdown
html
links
Need fields and not text? Add a prompt and get structured data back.
Recipes
Common Python crawler jobs
Chunk docs for a knowledge base
Crawl your docs, split the markdown on headings and send chunks to your embeddings.
job = client.crawl(
url="https://docs.acme.com/",
items_limit=500,
whitelist_regexp="/docs/",
)
chunks = []
for item in job.job_items:
if item.status != "done":
continue
for part in item.content.split("\n## "):
chunks.append({"url": item.original_url, "text": part})Building a chatbot? Use the LangChain loader or the MCP server.
Parameters
Python crawler settings: depth, URL filters, page limits
Change the settings and watch the Python code update.
max_depth
output_formats
whitelist_regexp
Your Python code
from webcrawlerapi import WebCrawlerAPI
client = WebCrawlerAPI(api_key="YOUR_API_KEY")
job = client.crawl(
url="https://docs.acme.com/",
items_limit=100,
output_formats=["markdown"],
max_depth=2,
)
print(job.status, len(job.job_items))| Argument | What it does |
|---|---|
| url | Seed URL where the crawler starts. |
| items_limit | Stops the crawl when it reaches this number of pages. Default 10. |
| max_depth | How many links deep to follow. 0 means the seed page only. |
| whitelist_regexp | Only URLs matching this pattern are crawled. |
| blacklist_regexp | URLs matching this pattern are skipped. |
| output_formats | Any of markdown, html, links. Default markdown. |
| respect_robots_txt | Skip pages disallowed by robots.txt. Default False. |
| max_age | Max cache age in seconds. 0 always fetches fresh. |
| webhook_url | Receive a POST when the job finishes. |
Results are cached for 7 days by default. See the caching docs.
SDK
The webcrawlerapi Python package
PyPI
pip install webcrawlerapi
GitHub
MIT licensed. Source, issues and examples.
Docs
Every method, parameter and response field.
| Method | What it does |
|---|---|
| crawl() | Crawl a site and wait until the job is done. |
| crawl_async() | Start a crawl and get the job id right away. |
| get_job() | Read job status and the crawled items. |
| cancel_job() | Stop a running job. Pages not started are not charged. |
| crawl_raw_markdown() | Crawl a site and get one combined markdown string. |
| scrape() | Scrape one URL and wait for the result. |
| scrape_async() / get_scrape() | Start a single-page scrape and read it later. |
| run_agent() | Run a crawling agent on a task described in plain text. |
Not on Python? There are also JavaScript, PHP and Java SDKs, and n8n, Make and Zapier integrations.
By the numbers
Reliable infrastructure, proven in production.
Tracking usage, crawl success rates, average response times, uptime percentages, quality scores, cost metrics, API performance, reliability data, and live trends for all your requests.
Active builders
Teams building in production
Extraction quality
Success rate
Fast turnaround
Average crawling time
Platform stability
Uptime
Which one
Scrapy alternative: WebCrawlerAPI vs Scrapy vs BeautifulSoup
| WebCrawlerAPI | requests + BeautifulSoup | Scrapy | |
|---|---|---|---|
| Setup | pip install + API key | pip install + selectors | Framework, middleware, deploy |
| Follows links | Yes, built in | You write it | Yes, you write spiders |
| JavaScript pages | Rendered for you | No | Needs a browser plugin |
| Retries | Handled for you | You maintain them | You maintain them |
| Output | Clean markdown / HTML | Raw HTML you parse | Whatever your spider yields |
| Best for | Content for apps and LLMs | One-off known pages | Huge custom pipelines |
Need one page and not a whole site? Use the Python web scraper. Using Node.js? See the JavaScript web crawler. Comparing providers? Read all crawler APIs compared.
Built for JavaScript-heavy sites
Per-site concurrency is shared and capped, so your crawl will not hammer the target site. See rate limits.
Pricing
Simple, transparent pricing
Start with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.
Pay as you go
No monthly commitment — pay only for what you use
$0/month
- From $0.002 / page
- LLM-cleaned markdown: $0.006 / page
- Up to 5 parallel requests
- Pay only for successful requests
- Webcrawler agent
Professional
Best for growing teams
$99/month
- From $0.0015 / page
- LLM-cleaned markdown: $0.0055 / page
- Up to 20 parallel requests
- Pay only for successful requests
- Webcrawler agent
Business
For high-volume crawling
$499/month
- From $0.001 / page
- LLM-cleaned markdown: $0.005 / page
- Up to 50 parallel requests
- Pay only for successful requests
- Webcrawler agent
Python web crawler FAQ
Common questions about crawling websites with Python
How do I build a web crawler in Python?
Run pip install webcrawlerapi, create a WebCrawlerAPI client with your API key and call client.crawl(url=..., items_limit=...). It follows links from the seed URL and returns a job with one item per page. No Scrapy spiders or headless browser setup.
Is this a Python library or an API?
Both. The webcrawlerapi package is a thin Python SDK over the WebCrawlerAPI REST API. The crawling, JavaScript rendering and retries run on our side, so your Python process only sends a request and reads results.
How is it different from BeautifulSoup and requests?
requests downloads one HTML file and BeautifulSoup parses it. You still write the link following, retries, JavaScript rendering and content cleaning yourself. WebCrawlerAPI does all of that and returns markdown.
How is it different from Scrapy?
Scrapy is a framework you run and scale yourself, including middleware and a browser plugin for JavaScript pages. WebCrawlerAPI is a hosted service: you call crawl() and get the pages back. Use Scrapy when you want full control of every request, and the API when you want the content.
Does it work with JavaScript-rendered pages?
Yes. Pages are rendered in a real browser on our infrastructure, so client-rendered content, lazy loading and cookie walls are handled. You do not install Selenium or Playwright.
Is there an async version for FastAPI?
The SDK is synchronous. For FastAPI or other async apps, use crawl_async() to start a job and return right away, then read the result with get_job() or receive it on a webhook_url. In async code you can also run crawl() in a worker thread.
Which Python versions are supported?
Python 3.6 and newer. The only dependency is requests.
How do I limit which pages get crawled?
Use whitelist_regexp and blacklist_regexp for URL patterns, max_depth for link depth, items_limit for the page count, and respect_robots_txt to follow the site robots.txt rules.
How much does it cost?
You pay per successfully crawled page: $0.002 on Pay as you go, $0.0015 on Professional and $0.001 on Business. Failed pages are not charged. Some sites are priced separately, see the pricing docs.
Can I try it before paying?
Yes. Sign up in the dashboard and get an API key. No credit card required.
Crawl your first site from Python in five minutes
Install the package, paste the example above and get markdown for every page. Or try it in the playground.