How do you clean and validate scraped data?

Scraped data fails silently. A selector that grabs the wrong element, a currency symbol left in a number field, or a date in one format on Monday and another on Tuesday — none of these throw an error, they just corrupt the record. Cleaning and validation exist to catch that before it reaches storage.

Common mistake

Writing extracted fields directly to storage with no type conversion or checks:

def scrape_product(url):
    soup = BeautifulSoup(requests.get(url).text, "html.parser")
    return {
        "name": soup.select_one(".title").text,
        "price": soup.select_one(".price").text,   # "$1,299.00" — a string, not a number
        "rating": soup.select_one(".rating").text,  # "4.5 out of 5 stars" — unparsed
    }

db.insert(scrape_product(url))

price stored as "$1,299.00" can't be sorted, filtered, or aggregated numerically. If the .rating selector ever returns None because the page has no reviews yet, this throws a NoneType error deep in production instead of failing the one field cleanly.

The fix

Define an explicit schema, coerce types deliberately, and handle missing fields as a valid state rather than a crash:

from pydantic import BaseModel, field_validator
from typing import Optional
import re

class Product(BaseModel):
    name: str
    price: float
    rating: Optional[float] = None

    @field_validator("price", mode="before")
    @classmethod
    def parse_price(cls, v):
        if isinstance(v, str):
            return float(re.sub(r"[^\d.]", "", v))
        return v

    @field_validator("rating", mode="before")
    @classmethod
    def parse_rating(cls, v):
        if v is None:
            return None
        match = re.search(r"[\d.]+", v)
        return float(match.group()) if match else None

def scrape_product(url) -> Product:
    soup = BeautifulSoup(requests.get(url).text, "html.parser")
    rating_el = soup.select_one(".rating")
    return Product(
        name=soup.select_one(".title").text.strip(),
        price=soup.select_one(".price").text,
        rating=rating_el.text if rating_el else None,
    )

Why it works

The schema makes "what does a valid record look like" explicit and enforced at the point of extraction, not discovered later when a downstream query breaks on a string where a number was expected. The validators isolate parsing logic per field — a currency format change only touches parse_price, not the whole extraction function. Making rating Optional turns a missing element from a crash into a legitimate, queryable state (rating IS NULL), which is what a missing rating actually means.

Tips

• Keep the raw scraped strings alongside the parsed values during a rollout — when a validator raises on unexpected input, you need the original text to fix the regex.
• Log validation failures with the source URL, not just the field name — you'll need to revisit that specific page to see what changed.
• Deduplicate on a stable key (URL, SKU) rather than on the full record — a single field change (price update) shouldn't register as a "new" item.
• Format choice affects how much cleaning you need downstream — see what data format works best for scraped data.

WebCrawlerAPI returns pre-cleaned markdown and structured JSON with boilerplate already stripped, so validation only has to check your own business rules, not fix extraction noise.