What data does a web crawler collect?

A crawler's job is discovery, so what it collects per page is metadata about the page — not the specific fields a scraper would extract. Getting this schema wrong shows up later as "we crawled the whole site but can't tell what changed" or "we have no way to deduplicate these two URLs."

Common mistake

Storing only the raw HTML per page, with no structured metadata:

def crawl_page(url):
    html = requests.get(url).text
    save_to_disk(url, html)  # just the raw bytes, nothing else

With just raw HTML on disk, answering "which pages redirected," "which pages 404'd," or "did this page change since last crawl" all require re-parsing every file from scratch — none of that information was captured at crawl time when it was cheap to record.

The fix

Capture a consistent per-page record alongside the content, with the fields that make later analysis and deduplication possible:

import hashlib
from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class CrawledPage:
    url: str
    final_url: str          # after following redirects
    status_code: int
    content_type: str
    content_hash: str       # for change detection and dedup
    crawled_at: str
    links_found: list = field(default_factory=list)

def crawl_page(url) -> CrawledPage:
    resp = requests.get(url, allow_redirects=True, timeout=10)
    content_hash = hashlib.sha256(resp.content).hexdigest()
    links = []
    if "text/html" in resp.headers.get("Content-Type", ""):
        soup = BeautifulSoup(resp.text, "html.parser")
        links = [urljoin(url, a["href"]) for a in soup.select("a[href]")]

    return CrawledPage(
        url=url,
        final_url=resp.url,
        status_code=resp.status_code,
        content_type=resp.headers.get("Content-Type", ""),
        content_hash=content_hash,
        crawled_at=datetime.utcnow().isoformat(),
        links_found=links,
    )

Why it works

Capturing final_url separately from the requested url makes redirect chains visible and lets you deduplicate two different starting URLs that resolve to the same canonical page. content_hash turns "did this page change since last time" into a single comparison instead of a byte-by-byte diff of raw HTML. links_found recorded at crawl time means you can rebuild the site's link graph later without re-crawling, which matters when you want to analyze internal linking structure or find orphaned pages after the fact.

Tips

• Store status_code even for successful requests — a crawl history full of 200s that suddenly shows a spike in 404s or 503s is often the first signal something changed on the target site.
• Content hashing is cheap and should happen on every crawl, even if you don't act on the result immediately — it's what makes freshness scheduling possible later; see how often to crawl a site.
• Keep links_found even for pages outside your crawl scope — a page linking heavily off-domain is useful signal for prioritization even if you don't follow those links.

WebCrawlerAPI returns this metadata — status, redirects, content, and discovered links — as structured output per page, so you don't have to define and maintain this schema yourself.