Python SDK

Python web crawler: crawl any website in 5 lines of code

pip install webcrawlerapi and get every page of a site as clean markdown. No headless browsers, no Scrapy spiders to maintain. You only pay for pages that succeed.

https://
  • ✓ Python 3.6+
  • ✓ Only dependency: requests
  • ✓ No credit card to try
  • ✓ Failed pages are free

crawler.py

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="YOUR_API_KEY")

job = client.crawl(
    url="https://docs.acme.com/",
    output_formats=["markdown"],
    items_limit=50,
    max_depth=2,
)

print(f"{job.status}: {len(job.job_items)} pages")

for item in job.job_items:
    if item.status == "done":
        print(item.original_url)
        print(item.content[:200])

One request, whole site

0 / 10 pages

//docs/help/blog/api/guides/faq/setup/post-1/post-2

seed URL → follows links → every page back as markdown

job 23b81e21 · running

Quickstart

How to crawl a website with Python

  1. 1. Install the SDK

    pip install webcrawlerapi
  2. 2. Get an API key

    Sign up and copy the key from the dashboard. No credit card needed to try it.
  3. 3. Call crawl()

    Pass a seed URL and items_limit. Loop over job.job_items and read item.content.

The problem

Why a DIY Python crawler wastes your month

requests gets an empty shell

Client-rendered pages return almost no HTML. Now you need Selenium or Playwright, and a browser to keep alive.

The crawler is the easy 10%

Link following, dedup, retries and rate limits grow into a project that never ends.

Parsed HTML is not LLM-ready

Menus and footers pollute embeddings. BeautifulSoup selectors break on every site redesign.

Prefer to build it yourself first? Read how to crawl a website with Python or should you build a crawler or use a crawling API.

The output

Pages in, clean markdown out

item.content is the page as markdown. No BeautifulSoup selectors, no boilerplate in your embeddings.

The page: docs.acme.com/api/authentication

Cookie banner: "We use cookies…"
Top navigation: Product · Pricing · Docs · Login
Sidebar menu (40 links)

Authentication

Every request needs an API key in the Authorization header. Create a key in the dashboard…

The part you want

Newsletter signup · Related posts · Footer (60 links)

What the crawl API returns (markdown)

# Authentication

Every request needs an API key in the
Authorization header.

## Create a key

1. Open **Dashboard → Access keys**
2. Click **New key**
3. Copy it. It is shown once.

```bash
curl -H "Authorization: Bearer $KEY" \
  https://api.acme.com/v1/me
```

Code

Python examples you can paste and run

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="YOUR_API_KEY")

job = client.crawl(
    url="https://docs.acme.com/",
    output_formats=["markdown"],
    items_limit=50,
    max_depth=2,
)

print(f"{job.status}: {len(job.job_items)} pages")

for item in job.job_items:
    if item.status == "done":
        print(item.original_url)
        print(item.content[:200])

What job and item give you

job.id                         # "23b81e21-c672-..."
job.status                     # new | in_progress | done | error | cancelled
job.job_items                  # list of JobItem
job.recommended_pull_delay_ms  # polling interval hint

item = job.job_items[0]
item.original_url              # "https://docs.acme.com/api/authentication"
item.title                     # "Authentication | Acme Docs"
item.page_status_code          # 200
item.status                    # "done"
item.content                   # page content, best available format
item.get_markdown()            # fetch markdown
item.get_html()                # fetch raw HTML

Full reference in the Python SDK docs, or read the source on GitHub. Prefer plain HTTP? See the crawl API page for curl.

Formats

Output formats for every page

markdown

Default. Headings, links and lists kept. Best for LLM context and search indexes.

html

The final HTML after JavaScript ran. Feed it to BeautifulSoup or your own parser.

links

Every link found on the page, for building URL lists.

Need fields and not text? Add a prompt and get structured data back.

Recipes

Common Python crawler jobs

Chunk docs for a knowledge base

Crawl your docs, split the markdown on headings and send chunks to your embeddings.

job = client.crawl(
    url="https://docs.acme.com/",
    items_limit=500,
    whitelist_regexp="/docs/",
)

chunks = []
for item in job.job_items:
    if item.status != "done":
        continue
    for part in item.content.split("\n## "):
        chunks.append({"url": item.original_url, "text": part})

Building a chatbot? Use the LangChain loader or the MCP server.

Parameters

Python crawler settings: depth, URL filters, page limits

Change the settings and watch the Python code update.

max_depth

output_formats

whitelist_regexp

Worst case for this job on Professional: 100 × $0.0015 = $0.15. You only pay for pages that succeed.

Your Python code

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="YOUR_API_KEY")

job = client.crawl(
    url="https://docs.acme.com/",
    items_limit=100,
    output_formats=["markdown"],
    max_depth=2,
)

print(job.status, len(job.job_items))
ArgumentWhat it does
urlSeed URL where the crawler starts.
items_limitStops the crawl when it reaches this number of pages. Default 10.
max_depthHow many links deep to follow. 0 means the seed page only.
whitelist_regexpOnly URLs matching this pattern are crawled.
blacklist_regexpURLs matching this pattern are skipped.
output_formatsAny of markdown, html, links. Default markdown.
respect_robots_txtSkip pages disallowed by robots.txt. Default False.
max_ageMax cache age in seconds. 0 always fetches fresh.
webhook_urlReceive a POST when the job finishes.

Results are cached for 7 days by default. See the caching docs.

SDK

The webcrawlerapi Python package

MethodWhat it does
crawl()Crawl a site and wait until the job is done.
crawl_async()Start a crawl and get the job id right away.
get_job()Read job status and the crawled items.
cancel_job()Stop a running job. Pages not started are not charged.
crawl_raw_markdown()Crawl a site and get one combined markdown string.
scrape()Scrape one URL and wait for the result.
scrape_async() / get_scrape()Start a single-page scrape and read it later.
run_agent()Run a crawling agent on a task described in plain text.

Not on Python? There are also JavaScript, PHP and Java SDKs, and n8n, Make and Zapier integrations.

By the numbers

Reliable infrastructure, proven in production.

Tracking usage, crawl success rates, average response times, uptime percentages, quality scores, cost metrics, API performance, reliability data, and live trends for all your requests.

Active builders

100+

Teams building in production

Extraction quality

98%

Success rate

Fast turnaround

2.5s

Average crawling time

Platform stability

99.98%

Uptime

Which one

Scrapy alternative: WebCrawlerAPI vs Scrapy vs BeautifulSoup

WebCrawlerAPIrequests + BeautifulSoupScrapy
Setuppip install + API keypip install + selectorsFramework, middleware, deploy
Follows linksYes, built inYou write itYes, you write spiders
JavaScript pagesRendered for youNoNeeds a browser plugin
RetriesHandled for youYou maintain themYou maintain them
OutputClean markdown / HTMLRaw HTML you parseWhatever your spider yields
Best forContent for apps and LLMsOne-off known pagesHuge custom pipelines

Need one page and not a whole site? Use the Python web scraper. Using Node.js? See the JavaScript web crawler. Comparing providers? Read all crawler APIs compared.

Built for JavaScript-heavy sites

Handled automatically
Automatic retries
Rate limit handling
Real headless browsers
JavaScript rendering
Smart caching
Data cleaning

Per-site concurrency is shared and capped, so your crawl will not hammer the target site. See rate limits.

Pricing

Simple, transparent pricing

Start with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.

Pay as you go

No monthly commitment — pay only for what you use

$0/month

Professional

Best for growing teams

Save 25%

$99/month

Business

For high-volume crawling

Save 50%

$499/month

Need a custom quota or special pricing? Contact us and we'll set up a plan that fits your needs.

See full pricing, limits and examples →

Python web crawler FAQ

Common questions about crawling websites with Python

How do I build a web crawler in Python?

Run pip install webcrawlerapi, create a WebCrawlerAPI client with your API key and call client.crawl(url=..., items_limit=...). It follows links from the seed URL and returns a job with one item per page. No Scrapy spiders or headless browser setup.

Is this a Python library or an API?

Both. The webcrawlerapi package is a thin Python SDK over the WebCrawlerAPI REST API. The crawling, JavaScript rendering and retries run on our side, so your Python process only sends a request and reads results.

How is it different from BeautifulSoup and requests?

requests downloads one HTML file and BeautifulSoup parses it. You still write the link following, retries, JavaScript rendering and content cleaning yourself. WebCrawlerAPI does all of that and returns markdown.

How is it different from Scrapy?

Scrapy is a framework you run and scale yourself, including middleware and a browser plugin for JavaScript pages. WebCrawlerAPI is a hosted service: you call crawl() and get the pages back. Use Scrapy when you want full control of every request, and the API when you want the content.

Does it work with JavaScript-rendered pages?

Yes. Pages are rendered in a real browser on our infrastructure, so client-rendered content, lazy loading and cookie walls are handled. You do not install Selenium or Playwright.

Is there an async version for FastAPI?

The SDK is synchronous. For FastAPI or other async apps, use crawl_async() to start a job and return right away, then read the result with get_job() or receive it on a webhook_url. In async code you can also run crawl() in a worker thread.

Which Python versions are supported?

Python 3.6 and newer. The only dependency is requests.

How do I limit which pages get crawled?

Use whitelist_regexp and blacklist_regexp for URL patterns, max_depth for link depth, items_limit for the page count, and respect_robots_txt to follow the site robots.txt rules.

How much does it cost?

You pay per successfully crawled page: $0.002 on Pay as you go, $0.0015 on Professional and $0.001 on Business. Failed pages are not charged. Some sites are priced separately, see the pricing docs.

Can I try it before paying?

Yes. Sign up in the dashboard and get an API key. No credit card required.

Crawl your first site from Python in five minutes

Install the package, paste the example above and get markdown for every page. Or try it in the playground.

Get your API key