How is web scraping different from web crawling?

Crawling answers "what pages exist." Scraping answers "what's in them." A crawler walks links to build a URL list; a scraper takes a URL and pulls out the fields you actually want. They're often chained in the same pipeline, but conflating them is where a lot of scraping code goes wrong.

Common mistake

Writing one function that both discovers pages and extracts data, tightly coupled:

def crawl_and_scrape(start_url, visited=None):
    visited = visited or set()
    if start_url in visited:
        return []
    visited.add(start_url)

    html = requests.get(start_url).text
    soup = BeautifulSoup(html, "html.parser")

    # Extraction logic buried inside the traversal loop
    results = [{
        "url": start_url,
        "title": soup.select_one("h1").get_text(strip=True) if soup.select_one("h1") else None,
    }]

    for link in soup.select("a[href]"):
        results += crawl_and_scrape(urljoin(start_url, link["href"]), visited)

    return results

This works for a small site, but it means you can't re-run extraction without re-crawling, can't change your parsing logic without touching traversal code, and can't parallelize the two stages independently.

The fix

Separate discovery from extraction into two passes with a URL list as the interface:

def crawl(start_url, max_pages=500):
    """Discovery: returns a list of URLs, nothing else."""
    visited, queue, urls = set(), [start_url], []
    while queue and len(urls) < max_pages:
        url = queue.pop(0)
        if url in visited:
            continue
        visited.add(url)
        html = requests.get(url).text
        urls.append(url)
        soup = BeautifulSoup(html, "html.parser")
        queue += [urljoin(url, a["href"]) for a in soup.select("a[href]")]
    return urls

def scrape(url):
    """Extraction: takes one URL, returns one structured record."""
    soup = BeautifulSoup(requests.get(url).text, "html.parser")
    title_el = soup.select_one("h1")
    return {"url": url, "title": title_el.get_text(strip=True) if title_el else None}

url_list = crawl("https://example.com")
records = [scrape(u) for u in url_list]

Why it works

With discovery and extraction as separate functions connected by a plain list of URLs, you can cache the crawl result and iterate on extraction logic without re-fetching the whole site, run extraction in parallel across workers since each scrape() call is independent, and swap either half (a different crawl strategy, a different parser) without touching the other.

Tips

• If you're building a one-off scrape of a known set of pages, you don't need a crawler at all — just a URL list and scrape().
• If you're building a scraper that runs on a schedule, the crawl stage is where you decide freshness and coverage; see how often you should crawl a site.
• Store the URL list from the crawl stage — it's a cheap, reusable artifact that lets you re-run extraction after fixing a selector bug, instead of re-crawling from scratch.

WebCrawlerAPI separates these concerns for you: a crawl job returns the URL list and rendered content for a whole site, so your code only has to handle the extraction logic on top.