What is web scraping?

Web scraping is the process of pulling specific fields out of a web page — a price, a title, a review count — and turning them into structured data you can store, query, or feed to another system. It sits downstream of discovery: something has to fetch the page first, but scraping is the part that decides what on that page actually matters.

Common mistake

The naive approach treats the whole page as the payload:

import requests

response = requests.get("https://example.com/product/123")
with open("product_123.html", "w") as f:
    f.write(response.text)

This saves everything — nav bars, ads, cookie banners, the actual product data — as one undifferentiated blob. Nothing downstream can query it, and re-scraping the same page a week later gives you no way to diff what changed.

The fix

Target the fields you need with a selector-based extraction, and shape the output as a schema from the start:

import requests
from bs4 import BeautifulSoup
from dataclasses import dataclass, asdict
import json

@dataclass
class Product:
    name: str
    price: float
    in_stock: bool

def scrape_product(url: str) -> Product:
    html = requests.get(url).text
    soup = BeautifulSoup(html, "html.parser")

    name = soup.select_one("h1.product-title").get_text(strip=True)
    price_text = soup.select_one("span.price").get_text(strip=True)
    price = float(price_text.replace("$", "").replace(",", ""))
    in_stock = soup.select_one(".stock-status").get_text(strip=True) == "In Stock"

    return Product(name=name, price=price, in_stock=in_stock)

product = scrape_product("https://example.com/product/123")
print(json.dumps(asdict(product)))

Why it works

Defining the schema (Product) before writing the selectors forces you to decide what "done" looks like — a record with three typed fields — instead of an open-ended HTML dump. Selectors targeting semantic markers (.product-title, .price) are also more stable across redesigns than positional selectors like div:nth-child(4), since class names tied to meaning tend to survive layout changes that reorder elements.

Tips

• Scrape into a schema, not a file. Even a simple dataclass or JSON shape makes downstream storage, deduplication, and change-detection trivial.
• Keep the raw HTML alongside the parsed record during development — when a selector breaks, you need the original markup to see why.
• Scraping is one half of a pipeline. Something has to find the URLs first — see how web scraping differs from web crawling for where that boundary sits.

If you'd rather skip writing and maintaining selectors, WebCrawlerAPI fetches and renders the page for you and returns clean, structured markdown or JSON — you point it at a URL instead of a parser.