Web Scraping with Python: A Practical Guide (Requests to Playwright)

Web scraping with Python usually starts the same way. A requests plus BeautifulSoup script is written, it works in five minutes, and it feels great. Then it is pointed at a second site, and the list comes back empty, or the server answers 403.

This guide scrapes one practice site with four approaches, in order, and gives a rule for when to move to the next one. It covers how to scrape a website with Python, save the data, survive errors, handle JavaScript pages, and tell when a home-made script stops being worth it.

Scraping means pulling fields out of pages you already know about (price, title, author). Crawling means discovering and following links across a site. If you need the second one, read the difference between web crawling and scraping or jump to the Python crawling guide. Prefer another language? There's a Node.js version of this guide.

Everything below was run in October 2026 on Python 3.11 with requests 2.34, BeautifulSoup 4.15, httpx 0.28 and Playwright 1.64. All outputs shown are real.

The short version: a ladder of four tools

1. requests + BeautifulSoup. Static HTML. Start here, always.
2. httpx + asyncio. Same thing, many pages at once. Only when sequential is too slow.
3. Playwright. The page needs JavaScript to show its data.
4. A scraping API. You're blocked, or the browser and proxy work costs more than the data is worth.

Move up a rung only when a concrete symptom appears: an empty result, a 403 or 429, or a job that takes hours. Most scrapers never need rung 3 or 4.

What to Check Before You Write Any Code

Spend 30 seconds on the target page first. It saves hours.

1. Is the data in the raw HTML? Open the page, right-click, choose "View page source", and search for a piece of text you want. If it's there, requests will work. If it's missing, the page builds itself with JavaScript (more on that below).

2. Is there a JSON endpoint? Open DevTools, go to the Network tab, filter by Fetch/XHR, and reload. Many "dynamic" pages just call a JSON URL. Calling it directly is faster and cleaner than any parser.

3. Does the site have an official API? Use it. It's stable, and nobody will be angry.

4. What do robots.txt and the terms say? A longer piece on whether web scraping is legal covers this. The short version: scraping public pages is generally lawful in the US, but terms of service, personal data and copyright can still cause trouble. This is not legal advice.

robots.txt can be checked from the standard library:

from urllib.robotparser import RobotFileParser

BASE = "https://quotes.toscrape.com"
USER_AGENT = "my-scraper/1.0 (me@example.com)"

rp = RobotFileParser()
rp.set_url(f"{BASE}/robots.txt")
rp.read()

for path in ["/", "/page/2/", "/js/"]:
    print(path, "allowed" if rp.can_fetch(USER_AGENT, f"{BASE}{path}") else "blocked")

Output:

/ allowed
/page/2/ allowed
/js/ allowed

Our Practice Target

The practice site is quotes.toscrape.com. It exists so people can practice scraping on it. It has:

• 100 quotes spread over 10 pages (/page/2/, /page/3/, ...)
• a JavaScript version at /js/, where the quotes are drawn by a script
• no login, no blocking

The same target is used for every step, so every number can be reproduced. Swap in your own site once the pattern makes sense.

Setup

Python 3.11 or newer is needed.

python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml httpx playwright
playwright install chromium

lxml is a fast HTML parser BeautifulSoup can use. playwright install chromium downloads a Chromium browser. Skip it for now if you only want the static steps.

Step 1: Fetch a Page with Requests

Three habits make a request script survive the real world:

• Send a User-Agent that says who you are (and how to reach you).
• Call raise_for_status() so a 404 or 500 doesn't pass as a page.
• Always set a timeout. Without one, a single stuck connection freezes the whole script forever. requests has no default timeout.

import requests

URL = "https://quotes.toscrape.com/"
HEADERS = {"User-Agent": "my-scraper/1.0 (me@example.com)"}

response = requests.get(URL, headers=HEADERS, timeout=10)
response.raise_for_status()
print(response.status_code, len(response.text), "bytes of HTML")

200 11021 bytes of HTML

If response.text shows garbled characters, the encoding was guessed wrong. Try response.encoding = response.apparent_encoding before reading .text.

Step 2: Parse It with BeautifulSoup

BeautifulSoup turns an HTML string into a tree you can search. CSS selectors (select, select_one) are used here instead of find_all. They're shorter, and they can be copied straight from DevTools ("Copy selector").

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "lxml")

rows = []
for quote in soup.select("div.quote"):
    author_node = quote.select_one("small.author")
    rows.append({
        "text": quote.select_one("span.text").get_text(strip=True),
        "author": author_node.get_text(strip=True) if author_node else None,
        "tags": [a.get_text(strip=True) for a in quote.select("a.tag")],
    })

print(len(rows), "quotes")
print(rows[0])

10 quotes
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}

Handle missing elements

The most common beginner error is AttributeError: 'NoneType' object has no attribute 'get_text'. It means select_one found nothing and returned None. Real pages have missing fields all the time (no price, no author). Check for None before calling a method on the result, like the author_node line above.

html.parser or lxml?

html.parser ships with Python and needs no install. lxml is faster and forgives broken HTML better, so it is the default here. The difference only matters at thousands of pages or on messy markup.

If your scraped text is full of leftover tags, whitespace and noise, see cleaning crawled data with BeautifulSoup. For a BeautifulSoup script that follows links, there's this crawler walkthrough.

Step 3: Pagination

One page is a toy. Real data sits on page 2, 3, 4 and so on. There are two ways to walk through it:

• Follow the "next" link until there isn't one. This is the safest. The site tells you when it's over.
• Count /page/N/ and stop at the first empty or 404 page. Works when the URL pattern is stable.

The first is preferred. Here's the loop. urljoin is used because "next" links are usually relative, and a sleep between requests keeps the server from being hammered.

import time
from urllib.parse import urljoin

def scrape_all(start_url: str) -> list[dict]:
    session = make_session()  # defined in Step 5
    rows, url, pages = [], start_url, 0
    while url:
        html = fetch(session, url)  # also defined in Step 5
        if html is None:
            break  # stop instead of looping forever on a dead page
        soup = BeautifulSoup(html, "lxml")
        rows.extend(parse_quotes(soup))  # the Step 2 selector code, wrapped in a function
        pages += 1
        print(f"page {pages}: {url} -> {len(rows)} quotes so far")

        next_link = soup.select_one("li.next > a")
        url = urljoin(url, next_link["href"]) if next_link else None
        time.sleep(1)  # be polite
    return rows

Real run:

10 pages, 100 quotes, about 12 seconds (mostly the 1-second sleeps). The end condition is explicit: no "next" link, loop ends. For more patterns (infinite scroll, "load more" buttons, cursor pagination), see this short note on handling pagination when scraping.

Step 4: Save the Data (CSV and JSON)

Printing rows is fine for learning. To use the data, write it to a file. CSV opens in Excel. JSON keeps nested data like lists.

import csv
import json

with open("quotes.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["text", "author", "tags"])
    writer.writeheader()
    writer.writerows(rows)

with open("quotes.json", "w", encoding="utf-8") as f:
    json.dump(rows, f, ensure_ascii=False, indent=2)

Two details that bite people. Always pass encoding="utf-8", or Windows will pick something else and mangle the curly quotes. And ensure_ascii=False keeps real characters in JSON instead of “ escapes.

The first lines of quotes.csv:

text,author,tags
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”,Albert Einstein,"change,deep-thoughts,thinking,world"
"“It is our choices, Harry, that show what we truly are, far more than our abilities.”",J.K. Rowling,"abilities,choices"

Not sure which format fits? This note on the best data format for scraped data covers it.

Step 5: Make It Not Break (Python Requests Retry, Timeouts, Rate Limits)

A scraper that runs for 10 pages can ignore errors. One that runs for 10,000 pages will hit a timeout, a 502 and a 429 before lunch.

Python's requests can retry on its own. A Retry policy is mounted on a Session:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

HEADERS = {"User-Agent": "my-scraper/1.0 (me@example.com)"}


def make_session() -> requests.Session:
    retry = Retry(
        total=4,
        backoff_factor=1,  # waits 1s, 2s, 4s... between tries
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET"],
        respect_retry_after_header=True,  # honors Retry-After on 429/503
    )
    session = requests.Session()
    session.headers.update(HEADERS)
    session.mount("https://", HTTPAdapter(max_retries=retry))
    session.mount("http://", HTTPAdapter(max_retries=retry))
    return session


def fetch(session: requests.Session, url: str) -> str | None:
    try:
        response = session.get(url, timeout=10)
        response.raise_for_status()
        return response.text
    except requests.RequestException as error:
        print(f"FAILED {url}: {error}")
        return None

What this gives you:

• Retries with backoff on 429 and 5xx errors. Waiting longer each time is kind to a struggling server.
• Retry-After support. When a server says "come back in 30 seconds", the adapter listens.
• Failures are logged, not fatal. fetch returns None, and your loop decides what to do. One bad page doesn't kill a 4-hour job.
• A Session reuses the connection, so it's also faster than bare requests.get.

The failure path was tested against a URL that always returns 500. After the retries ran out, the script printed this and kept going:

FAILED https://httpbin.org/status/500: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /status/500 (Caused by ResponseError('too many 500 error responses'))

A 429 means "too many requests", and the fix is almost always to slow down. Here's more on what a 429 error means in web scraping. A fixed delay between requests (like the time.sleep(1) above) prevents most of them.

Step 6: Go Faster with httpx and asyncio

Sequential scraping waits for each response before asking for the next. Async sends several requests at once. For 10 pages this saves seconds. For 10,000 pages it saves hours.

import asyncio
import httpx
from bs4 import BeautifulSoup

URLS = [f"https://quotes.toscrape.com/page/{n}/" for n in range(1, 11)]
MAX_CONCURRENT = 3  # keep this low, it is a politeness limit


async def fetch(client, sem, url):
    async with sem:
        response = await client.get(url, timeout=10)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "lxml")
        return len(soup.select("div.quote"))


async def main():
    sem = asyncio.Semaphore(MAX_CONCURRENT)
    async with httpx.AsyncClient(headers=HEADERS) as client:
        counts = await asyncio.gather(*(fetch(client, sem, u) for u in URLS))
    print(sum(counts), "quotes")

asyncio.run(main())

The Semaphore caps how many requests run at the same time. Concurrency is a politeness problem as much as a speed trick. 3 parallel requests is friendly. 300 looks like an attack, and you'll be blocked fast.

The same 10-page job was run three times each way on a laptop (home connection, October 2026). Each run was a fresh process, and memory is the peak total of the script plus any browser it started. The Playwright and API rows read the JavaScript version of the site (/js/page/N/), because requests can't read it at all. The API was run with its cache turned off.

httpx is a small win on 10 small pages, and the win grows with page count. Playwright is where memory matters. One reused browser needs about 530 MB. Ten tabs at once finish fastest (1.4 seconds) but peak at 1.3 GB, and the peak grows with every tab. The API is slower per page, since each page is rendered by a remote browser, but local memory stays at about 50 MB whether 1 request or 10 are sent. It's not as fast as it sounds: 10 parallel requests took 12.5 seconds in these runs, with one slow outlier at 31.8 seconds. Concurrency is also capped (10 parallel requests per account, see rate limits). The tradeoff is time against local memory and ops work. The requests run has no sleep, unlike the polite loop in Step 3. Treat these numbers as one machine, one site, one afternoon.

When requests Stops Working: Dynamic Pages

Now the same quotes, but on quotes.toscrape.com/js/. The page looks identical in a browser. Here's what requests sees:

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://quotes.toscrape.com/js/",
    headers={"User-Agent": "my-scraper/1.0 (me@example.com)"},
    timeout=10,
)
print("status:", response.status_code)

soup = BeautifulSoup(response.text, "lxml")
print("quotes found:", len(soup.select("div.quote")))

Status 200, zero quotes. No error, no warning, just an empty list. That's the classic symptom. The server sent a nearly empty page plus a script, and the script draws the quotes in the browser. requests doesn't run scripts.

You have two options:

1. Find the data source. In DevTools, Network tab, look for a JSON request the page makes. If you find one, call it with requests and skip the browser entirely. This is the best outcome.
2. Render the page in a real browser and read the result.

For a deeper look at how rendering works, see the JavaScript rendering guide and this short note on scraping JavaScript-heavy sites.

Scrape JavaScript Pages with Playwright

Playwright controls a real Chromium browser from Python. It loads the page, runs the scripts, and exposes the final HTML. That HTML can then be passed to the same BeautifulSoup code from Step 2.

from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", timeout=30_000)
    # Wait for the element we need, not a fixed number of seconds
    page.wait_for_selector("div.quote", timeout=10_000)
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "lxml")
quotes = soup.select("div.quote")
print("quotes found:", len(quotes))
print(quotes[0].select_one("span.text").get_text(strip=True))

Zero became 10. A few notes from real use:

• Use wait_for_selector, not time.sleep. A fixed sleep is either too short (flaky) or too long (slow). Waiting for the element you need is both faster and more reliable.
• A browser is heavy. Launching Chromium for a single page took about 1.4 seconds, and 10 pages with one reused browser took 3.9 seconds and about 530 MB of memory at peak (see the table in Step 6). The requests version used 45 MB. Looping Playwright over thousands of pages needs real memory and real time.
• Reuse the browser. Launch once, open many pages. Don't launch per URL.
• Selectors break. If a redesign changes class names, your scraper returns nothing. See how to fix Playwright CSS class selector breakage.

When Playwright misbehaves, these are the most common errors: action timeout errors, the executable does not exist error (you forgot playwright install), and overusing waitForTimeout.

What about Selenium and Crawl4AI?

Selenium still works, and plenty of teams have test suites built on it. For new scraping code, Playwright is the better pick: auto-waiting is built in, and setup is a single playwright install. If you want an open-source tool that returns Markdown for LLM use, look at Crawl4AI. Scrapy is a full framework for big crawl pipelines, covered in this Scrapy tutorial.

When Your IP Gets Blocked: 403, 429, CAPTCHA

Scraping one practice site is polite and easy. Real sites push back. Here's how the signals can be read:

Start by respecting limits: read robots.txt, slow down, and identify yourself. Many 429s vanish with a 1-2 second delay. There's more in this note on avoiding blocks when scraping.

Here's the honest part. Beyond the basics, surviving heavy anti-bot systems means maintaining a proxy pool, rotating browser fingerprints, solving or avoiding CAPTCHAs, and re-fixing all of it when the site changes. That's a part-time job. You can do it. Many teams decide it isn't the job they want.

The API Way: Let Someone Else Handle Proxies and Blocks

A scraping API takes over the parts that hurt at scale: the proxy pool, retries and blocking. You send a URL and get the page back as Markdown, HTML or structured data. WebCrawlerAPI also renders JavaScript pages. Services differ here, so whatever you pick, test it on the page that gave you trouble first.

Here's the same /js/ page with the WebCrawlerAPI Python SDK:

pip install webcrawlerapi

import os
from webcrawlerapi import WebCrawlerAPI

client = WebCrawlerAPI(api_key=os.environ["WEBCRAWLERAPI_KEY"])

result = client.scrape(
    url="https://quotes.toscrape.com/js/",
    output_formats=["markdown"],
)

print("status code:", result.page_status_code)
print(result.markdown[:300])

Real output, cut to the first 300 characters:

No parsing code, no proxy setup, no retry logic on your side. The key comes from an environment variable so it never lands in your code.

If you want fields instead of text, pass a prompt and a response_schema and read result.structured_data. See structured outputs. Not using Python? The same call over HTTP (details in the scrape docs):

curl --request POST \
  --url https://api.webcrawlerapi.com/v2/scrape \
  --header "Authorization: Bearer $WEBCRAWLERAPI_KEY" \
  --data '{
  "url": "https://quotes.toscrape.com/js/",
  "output_formats": ["markdown"]
}'

When not to use an API

This post comes from an API vendor, so a bias is likely. Here is where an API isn't needed:

• A one-off script for a few static pages. requests is free and fine.
• Learning. The first scraper should be written by hand.
• A site with a clean JSON endpoint. Call it directly.

You pay per page and depend on a vendor. In return you save the engineering time for browsers, proxies and selector upkeep. If you're comparing vendors, here's a list of the best web scraping APIs. To try WebCrawlerAPI from Python, the Python scraper page has the quick start.

Requests vs Playwright vs Scrapy vs an API

One real page, three ways

Practice sites are friendly, so a real page was tried as well: a Dailymotion video page. The goal was to get the video description. Each approach was run five times (October 2026, same laptop, API cache bypassed with max_age=0):

requests received a 200 and the title, but the description only exists after the page renders, so it was never in the response. Playwright found it and used 13 times the memory of requests. Browser launch is fast on a modern laptop (about 0.3 seconds), and the page load is the noisy part. The API found it too, used almost no local memory, and was the slowest for a single uncached page (about 5.7 seconds). A repeat request for the same URL was served from the cache in about half a second.

So it isn't a speed win. What you buy is no browser on your machine, no proxy setup, and blocks handled by someone else. That matters when you run thousands of pages from a server, and much less for a single page on your laptop. One page on one site proves little, so run this test on your own target before choosing.

A rule of thumb: requests for anything static, Playwright when a page needs scripts, an API when blocks or volume turn scraping into an infrastructure project (and test it on your page first). Scrapy sits off to the side for teams that crawl whole sites on a schedule.

Common Errors and Fixes

The list comes back empty

The data is probably built by JavaScript. Run the "View page source" test. If your text isn't in the source, use Playwright or find the JSON endpoint.

'NoneType' object has no attribute 'get_text'

A selector matched nothing. Check the selector in DevTools, and guard optional fields with an if check.

403 Forbidden

Add a real User-Agent, slow down, and re-read the robots.txt. If it still fails from your IP, see the blocking section above.

Garbled characters

Set response.encoding = response.apparent_encoding, or write files with encoding="utf-8".

The script hangs forever

You forgot timeout=. Add it to every request.

It worked last month, now it returns nothing

The site changed its HTML. This is the main cost of owning a scraper. Pick selectors that depend on meaning (like [itemprop="price"] or a stable data- attribute) over generated class names where possible.

Is Web Scraping Legal?

Scraping publicly available data is generally allowed in the US, but that's a summary, not a guarantee. Terms of service, personal data rules like GDPR, and copyright can all apply, and courts have ruled both ways. Real cases are covered in the posts on scraping cases where scrapers won and cases where they lost. Talk to a lawyer for anything commercial or sensitive.

FAQ

Is Python good for web scraping?

Yes. The libraries are mature (requests, BeautifulSoup, Playwright, Scrapy), and the syntax stays out of your way. Python is also what most data and AI teams already use for the next step after scraping. A comparison with Node is in Python vs Node.js for web crawling.

What is the best Python library for web scraping?

There's no single best one. requests plus BeautifulSoup for static pages, Playwright for JavaScript pages, Scrapy for large crawls. Pick by page type, not by popularity.

Should I use BeautifulSoup or Scrapy?

BeautifulSoup parses HTML; it doesn't fetch, schedule or retry. Scrapy is a whole framework with queues, concurrency and pipelines. For a script that scrapes a handful of pages, BeautifulSoup is simpler. For a recurring crawl of thousands of pages, Scrapy saves you from rebuilding the plumbing.

BeautifulSoup or Playwright for dynamic pages?

Playwright, because BeautifulSoup can't run JavaScript. Use them together though: Playwright renders the page, BeautifulSoup parses the resulting HTML.

Do I still need requests if I use Playwright?

For static pages, yes. Requests is far lighter. Use the browser only on pages that need it.

How do I scrape a JavaScript website with Python?

First look for a JSON endpoint in the DevTools Network tab. If there isn't one, render the page with Playwright, wait for the element you need, and parse the HTML.

Can I scrape a website without getting blocked?

Often, if you're polite: respect robots.txt, send a real User-Agent, add delays, and keep concurrency low. Sites with strong bot protection need proxies and browser fingerprinting, which is where an API starts to pay off.

How do I save scraped data to CSV or JSON?

Use csv.DictWriter for CSV and json.dump(..., ensure_ascii=False) for JSON, both with encoding="utf-8". See Step 4.

Is web scraping illegal?

Not by default. Scraping public pages is generally allowed in the US, but terms of service, personal data rules and copyright can still apply. See the legal section above and the longer ethics guide. This isn't legal advice.

How can I get the HTML of a URL with Python?

requests.get(url, timeout=10).text returns the HTML as a string. Add a User-Agent header and call raise_for_status(), as in Step 1. If the string lacks the data you see in the browser, the page needs JavaScript, so use Playwright.

Can ChatGPT scrape websites?

Not on its own. A chat model can write the scraper or parse HTML you paste in, but something still has to fetch the pages. That is what the code in this guide does. Some APIs return Markdown that is easier to pass to a language model.

Is BeautifulSoup a Python library?

Yes. It's the beautifulsoup4 package, imported as bs4. It parses HTML and lets you pick elements with CSS selectors. It doesn't fetch pages, which is why it's paired with requests or httpx.

What's the difference between web scraping and web crawling?

Scraping extracts data from pages you already know. Crawling discovers pages by following links. Most real projects do both. More in the full comparison.

What to Do Next

The ladder again: requests + BeautifulSoup for static pages, httpx when you need speed, Playwright when JavaScript hides the data, and an API when blocks or scale take over your week.

Good next steps:

• To follow links across a whole site, read how to crawl a website with Python.
• To scrape without maintaining browsers and proxies, try the WebCrawlerAPI Python SDK. Sign up, grab a key, and run the scrape_api.py example above on a page that gives you trouble.
• To read about the legal side first, see Is Web Scraping Legal?

Start with the smallest tool that works. Move up when the page tells you to.