Python SDK

Python web scraper: scrape any website in one call

pip install webcrawlerapi and scrape JavaScript-rendered pages through one API as clean markdown, HTML or links. No Selenium, no proxies, no retries to maintain. You only pay for pages that succeed.

Get your API key Python SDK docs

GitHub PyPI

https://

✓ Python 3.6+
✓ Only dependency: requests
✓ No credit card to try
✓ Failed pages are free

scrape.py

result.markdown

waiting

The page is rendered in a real browser, then converted to markdown.

one request → rendered page → clean markdown

Trusted by 1,000+ developers building AI products and data pipelines

Quickstart

How to scrape a website with Python in 3 steps

1. Install the SDK
```
pip install webcrawlerapi
```
2. Get an API key
Sign up and copy the key from the dashboard. No credit card needed to try it.
3. Call scrape()
Pass a url and the output_formats you want. Read result.markdown.

The problem

Why scraping JavaScript sites in Python hurts

requests gets an empty shell

Single-page apps draw their content with JavaScript. requests.get() returns a <div id="root"> and nothing else.

Selenium is a second job

Browser and driver versions, memory leaks, crashed pages, Docker images that weigh a gigabyte. The scraper works on Monday and breaks by Friday.

Sites block bots

Rate limits, captchas and blocked IPs mean proxies and retries. That work never ends.

Weighing it up? Read should you build a crawler or use a crawling API.

The difference

requests vs headless browser: scraping JavaScript pages in Python

A plain request gets the HTML the server sent. The scraper API gets what a visitor sees after the JavaScript has run.

requests + BeautifulSoup

0 products

<!doctype html>
<html>
  <head><title>Acme Store</title></head>
  <body>
    <div id="root"></div>
    <script src="/static/app.4f9a.js"></script>
  </body>
</html>

The products are drawn by JavaScript after load. requests.get() never sees them.

client.scrape()

24 products

# Acme Store: Products

## Trail Runner X — $129
Lightweight shoe, 24 sizes in stock.

## Summit Jacket — $249
Waterproof shell, packs into its pocket.

…22 more products

Same URL, rendered in a real browser first. You get the content a visitor sees.

How it works

How the Python scraper works: one call, one page

1. Send a URL
Call client.scrape() with a url and the output_formats you want.
2. We render it
The page loads in a real browser. Retries, proxies and anti-bot handling run on our side.
3. Get the content back
The result has markdown, raw_content or links. No job to poll. Need async? Read about async requests.

Shape the output

Change the settings and watch the Python code update.

output_formats

clean_selectors

respect_robots_txt

prompt (structured data)

One call, one result. No job to poll. You pay only when the page scrapes successfully.

Your Python code

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="YOUR_API_KEY")

result = client.scrape(
    url="https://shop.acme.com/products",
    output_formats=["markdown"],
)

if result.success:
    print(result.markdown)

Argument	What it does
url	The page to scrape. Required.
output_formats	Any of markdown, html, links. Default markdown.
clean_selectors	CSS selectors to remove from the output.
prompt	Ask for specific fields. Returns structured_data. Billed per token.
response_schema	JSON Schema for the structured output. Use with prompt.
respect_robots_txt	Return an error if robots.txt disallows the URL. Default False.
max_age	Max cache age in seconds. 0 always fetches fresh.
webhook_url	Receive a POST when an async scrape finishes.

Full reference in the POST /scrape docs.

Code

Web scraping with Python: code examples

from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key="YOUR_API_KEY")

result = client.scrape(
    url="https://shop.acme.com/products",
    output_formats=["markdown", "links"],
)

if result.success:
    print(result.markdown)
    print(result.links)
else:
    print(f"Error: {result.error_code} - {result.error_message}")

What the result gives you

result.success           # True
result.status            # "done" | "error" | "in_progress"
result.markdown          # "# Acme Store: Products ..."
result.raw_content       # final HTML (if "html" requested)
result.links             # ["https://shop.acme.com/products/..."]
result.page_status_code  # 200
result.page_title        # "Acme Store: Products"
result.structured_data   # dict, if you passed a prompt

# On failure you get an error object instead
result.error_code        # machine-readable code
result.error_message     # human-readable message

Full reference in the Python SDK docs, or read the source on GitHub. Prefer plain HTTP? See the scrape API docs.

Formats

Output formats for every scraped page

markdown

Default. Headings, links and lists kept. Best for LLM context and search indexes.

html

The final HTML after JavaScript ran. Feed it to BeautifulSoup or your own parser.

links

Every link found on the page, for building URL lists.

Need fields and not text? Add a prompt and get structured data back as a Python dict.

Recipes

Common Python scraping jobs

Monitor prices

Ask for fields instead of writing selectors that break on every redesign.

result = client.scrape(
    url="https://shop.acme.com/p/trail-runner-x",
    prompt="Extract product name and price",
    response_schema={
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "price": {"type": "number"},
        },
    },
)

if result.success:
    save_price(result.structured_data)

def read_page(url: str) -> str:
    result = client.scrape(
        url=url,
        output_formats=["markdown"],
        clean_selectors="nav, footer, .cookie-banner",
    )
    return result.markdown if result.success else ""

context = read_page("https://docs.acme.com/api")

import pandas as pd

urls = ["https://acme.com/a", "https://acme.com/b"]
rows = []

for url in urls:
    r = client.scrape(url=url)
    if r.success:
        rows.append(
            {"url": url, "title": r.page_title, "chars": len(r.markdown)}
        )

df = pd.DataFrame(rows)

from fastapi import FastAPI
from webcrawlerapi import WebCrawlerAPI

app = FastAPI()
client = WebCrawlerAPI(api_key="YOUR_API_KEY")

@app.post("/scrape")
def start(url: str):
    return {"id": client.scrape_async(url=url).id}

@app.get("/scrape/{scrape_id}")
def read(scrape_id: str):
    r = client.get_scrape(scrape_id)
    return {"status": r.status}

Building an agent? Use the LangChain loader or the MCP server.

Use cases

What developers build with the Python scraper

Price and product monitoring

Before: prices load after the page does, so BeautifulSoup sees nothing.

After: scrape the rendered page, or ask for name and price fields directly.

prompt + response_schema

Context for AI agents

Before: your agent gets menus and cookie banners instead of the answer.

After: hand it clean markdown of exactly the page it asked for.

output_formats=["markdown"]

Data pipelines and notebooks

Before: a Selenium setup in every Jupyter environment and CI job.

After: one pip install and an API key. Works anywhere Python runs.

client.scrape(url=...)

SEO and competitor checks

Before: a headless browser farm just to read titles and links.

After: get HTML and links for any page from a cron job or Lambda.

output_formats=["html", "links"]

SDK

The webcrawlerapi Python package

PyPI

pip install webcrawlerapi

GitHub

MIT licensed. Source, issues and examples.

Docs

Every method, parameter and response field.

Method	What it does
scrape()	Scrape one URL and wait for the result.
scrape_async()	Start a scrape and get an id right away.
get_scrape()	Read the result of an async scrape by id.
crawl()	Crawl a whole site from one seed URL.
run_agent()	Run a crawling agent on a task described in plain text.

Need a whole site and not one page? See the Python web crawler. Not on Python? See the JavaScript web scraper for Node.js. There are also JavaScript, PHP and Java SDKs, and n8n, Make and Zapier integrations.

By the numbers

Reliable infrastructure, proven in production.

Tracking usage, crawl success rates, average response times, uptime percentages, quality scores, cost metrics, API performance, reliability data, and live trends for all your requests.

Active builders

100+

Teams building in production

Extraction quality

98%

Success rate

Fast turnaround

2.5s

Average crawling time

Platform stability

99.98%

Uptime

Which one

Selenium alternative: WebCrawlerAPI vs Selenium, Playwright and BeautifulSoup

	WebCrawlerAPI	Selenium / Playwright	requests + BeautifulSoup
Renders JavaScript	Yes	Yes	No
Setup	pip install, one API key	Install browsers and drivers, manage versions	pip install
Runs in serverless / CI	Yes, it is just an HTTP call	Hard, large binaries	Yes
Proxies and retries	Handled for you	You build them	You build them
Output	Markdown, HTML, links, structured data	Whatever you extract	Raw HTML you parse
Cost model	Per successful page	Servers and your time	Servers and your time

Comparing providers? Read all crawler APIs compared.

Built for sites that fight back

Handled automatically

Residential proxies

Automatic retries

Rate limit handling

Real headless browsers

JavaScript rendering

CAPTCHA solving

Anti-bot bypass

Fastest path selection

Skip browser maintenance. See rate limits.

Pricing

Simple, transparent pricing

Start with pay-per-request, or save with a monthly subscription. Top-up credits are always available when your included allowance runs out.

Pay as you go

No monthly commitment — pay only for what you use

$0/month

From $0.002 / page
LLM-cleaned markdown: $0.006 / page
Unlimited proxy included
Up to 5 parallel requests
Pay only for successful requests
Webcrawler agent

Get started

Starter

Great for small projects

$29/month

From $0.002 / page
LLM-cleaned markdown: $0.006 / page
Unlimited proxy included
Up to 10 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Professional

Best for growing teams

Save 25%

$99/month

From $0.0015 / page
LLM-cleaned markdown: $0.0055 / page
Unlimited proxy included
Up to 20 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Business

For high-volume crawling

Save 50%

$499/month

From $0.001 / page
LLM-cleaned markdown: $0.005 / page
Unlimited proxy included
Up to 50 parallel requests
Pay only for successful requests
Webcrawler agent

Try for free

Need a custom quota or special pricing? Contact us and we'll set up a plan that fits your needs.

Python scraper FAQ

Common questions about scraping websites from Python

What is a Python web scraper API?

A scraper API loads a page in a real browser, waits for the JavaScript to run, and returns the final content. With WebCrawlerAPI that is one client.scrape() call in Python, backed by POST /v2/scrape.

How do I scrape a website with Python?

Run pip install webcrawlerapi, create a WebCrawlerAPI client with your API key and call client.scrape(url=..., output_formats=["markdown"]). The result has a markdown field. No parsing code needed.

Can it scrape JavaScript-rendered pages, like React or Next.js sites?

Yes. The page is rendered in a real browser before the content is extracted, so content that requests.get() never sees (client-rendered lists, lazy-loaded sections) is part of the result.

Do I need Selenium, Playwright or Scrapy?

No. You send a URL and get the content back. Browsers, retries, proxies and anti-bot handling run on our side, so there is nothing to install, patch or scale.

How is it different from BeautifulSoup?

BeautifulSoup parses HTML you already downloaded. It cannot fetch JavaScript-rendered content and you write selectors per site. The scraper API returns the rendered page as markdown, or structured data if you add a prompt.

Which output formats can I get?

Set output_formats to any of markdown, html and links. Add a prompt (and optionally a response_schema) to get structured data back as a Python dict.

Is scrape() synchronous?

Yes. client.scrape() waits and returns the result in the same call. For long-running work or async apps use scrape_async() and get_scrape() to poll by id, or set a webhook_url.

Which Python versions are supported?

Python 3.6 and newer. The only dependency is requests.

How much does scraping cost?

You pay per successful page: $0.002 on Starter, $0.0015 on Professional and $0.001 on Business. Failed requests are not charged. Some sites are priced separately, see the pricing docs.

Can I scrape many pages, or a whole site?

scrape() is for a URL you already know. To discover and fetch many pages from one seed URL, use crawl() or see the Python web crawler page.

Does the scraper respect robots.txt?

Only if you pass respect_robots_txt=True. The default is False. If the URL is disallowed, you get an error back.

Can I try it before paying?

Yes. Use the playground or sign up in the dashboard. No credit card required.

Scrape your first page from Python in five minutes

Install the package, run the snippet above, and read the rendered page as markdown. Or try it in the playground.

Get your API key

Python web scraper: scrape any website in one call

How to scrape a website with Python in 3 steps

1. Install the SDK

2. Get an API key

3. Call scrape()

Why scraping JavaScript sites in Python hurts

requests gets an empty shell

Selenium is a second job

Sites block bots

requests vs headless browser: scraping JavaScript pages in Python

How the Python scraper works: one call, one page

1. Send a URL

2. We render it

3. Get the content back

Shape the output

Web scraping with Python: code examples

Output formats for every scraped page

markdown

html

links

Common Python scraping jobs

What developers build with the Python scraper

Price and product monitoring

Context for AI agents

Data pipelines and notebooks

SEO and competitor checks

The webcrawlerapi Python package

PyPI

GitHub

Docs

Reliable infrastructure, proven in production.

Selenium alternative: WebCrawlerAPI vs Selenium, Playwright and BeautifulSoup

Built for sites that fight back

Simple, transparent pricing

Pay as you go

Starter

Professional

Business

Python scraper FAQ

Scrape your first page from Python in five minutes