How is web crawling different from web scraping?

Crawling and scraping get conflated because most real pipelines do both — but they optimize for different things, and treating them as one problem tends to produce code that's hard to tune. Crawling is about coverage: which pages do you need. Scraping is about precision: what fields do you pull from each one.

Common mistake

Sizing a "crawler" purely by request throughput, ignoring that discovery and extraction have different cost profiles:

def crawl_site(seed):
    queue = [seed]
    visited = set()
    all_data = []
    while queue:
        url = queue.pop()
        if url in visited:
            continue
        visited.add(url)
        html = requests.get(url).text
        soup = BeautifulSoup(html, "html.parser")

        # Heavy extraction logic runs on every single page during traversal
        all_data.append(extract_full_product_details(soup))  # 15+ fields, several regexes

        queue += [urljoin(url, a["href"]) for a in soup.select("a[href]")]
    return all_data

If only 200 of the 5,000 crawled pages are actual product pages, this runs expensive extraction logic on 4,800 pages that don't need it — category pages, the about page, pagination links — wasting the majority of the compute budget on pages with nothing to extract.

The fix

Let discovery classify pages cheaply, and only run full extraction on the pages that match:

def crawl_and_classify(seed):
    queue, visited, candidates = [seed], set(), []
    while queue:
        url = queue.pop()
        if url in visited:
            continue
        visited.add(url)
        html = requests.get(url).text
        soup = BeautifulSoup(html, "html.parser")

        # Cheap check: is this a page worth scraping at all?
        if soup.select_one(".product-title"):
            candidates.append(url)

        queue += [urljoin(url, a["href"]) for a in soup.select("a[href]")]
    return candidates

def scrape_candidates(urls):
    return [extract_full_product_details(url) for url in urls]

candidate_urls = crawl_and_classify("https://example.com")
products = scrape_candidates(candidate_urls)

Why it works

A cheap classification check (.product-title exists) during the crawl pass filters out the pages that don't matter before paying the cost of full extraction, so the expensive logic only runs on the subset of pages worth it. Splitting the two passes also means you can re-run extraction on the candidate list after fixing a parsing bug without re-crawling the whole site — the URL list is a durable artifact between stages.

Tips

• Crawling scales with the number of pages on a site; scraping scales with how many fields you need per page. They hit different bottlenecks and should be tuned separately.
• A crawl's job is done once it has a URL list and enough signal to classify pages — leave field extraction to a dedicated scrape pass.
• If your site has a sitemap.xml, use it to shortcut discovery instead of crawling link-by-link — see what data a crawler collects for what to capture during that pass.

WebCrawlerAPI separates these stages for you — a crawl job returns the full page list and content, so your extraction logic only runs on the pages you actually care about.