What are common web crawling tools?

Crawling tools split on how they handle scale and scheduling, not on whether they can fetch a page. A tool built for one-off exploratory crawling and one built to sustain a scheduled crawl of millions of pages solve different problems, even though both technically "follow links."

Common mistake

Building a custom crawler from scratch for a job that needs to run repeatedly at scale:

def crawl(seed, max_pages=100_000):
    queue, visited, pages = [seed], set(), []
    while queue and len(pages) < max_pages:
        url = queue.pop(0)
        if url in visited:
            continue
        visited.add(url)
        html = requests.get(url).text  # no retry, no concurrency, no persistence
        pages.append(html)
        queue += extract_links(html)
    return pages

At small scale this works. At the scale the loop is written for (100,000 pages), it has no retry handling for transient failures, no persistence if the process crashes at page 80,000, and no concurrency — it's single-threaded against a job that will take hours.

The fix

Use a framework built for this — Scrapy for large-scale HTML crawling with built-in scheduling, retries, and persistence:

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = ["https://example.com"]

    custom_settings = {
        "CONCURRENT_REQUESTS_PER_DOMAIN": 4,
        "RETRY_TIMES": 3,
        "JOBDIR": "crawl_state",  # persists queue state, survives a restart
    }

    def parse(self, response):
        yield {"url": response.url, "status": response.status}
        for link in response.css("a::attr(href)").getall():
            yield response.follow(link, callback=self.parse)

For pages that require JavaScript rendering to reveal links (SPA route discovery), pair the crawl with a headless browser layer — Playwright or Puppeteer driving the traversal, or a Scrapy integration that renders before parsing.

Why it works

Scrapy's JOBDIR setting persists the crawl frontier to disk, so a crash at page 80,000 resumes from there instead of restarting at zero — the single biggest practical difference at scale versus a hand-rolled loop. Built-in retry and per-domain concurrency settings (RETRY_TIMES, CONCURRENT_REQUESTS_PER_DOMAIN) handle the transient-failure and pacing concerns that a custom script has to reimplement from scratch, and get exercised far more in production than in a quick test run.

Tips

• Apache Nutch is the heavier-duty option when crawling is paired with building a search index — it integrates with Hadoop/Solr for that use case specifically.
• Reach for Playwright or Puppeteer in the crawl layer only when links themselves are generated by client-side JavaScript — see crawling JavaScript-heavy sites; for static-HTML link discovery they're unnecessary overhead.
• Persisted crawl state (Scrapy's JOBDIR, or your own checkpointing) is the feature that matters most once a crawl job runs longer than a few minutes — prioritize it before concurrency tuning.
• Tool choice for extraction is a separate decision — see common scraping tools once you have the URL list.

If you'd rather not run and maintain crawl infrastructure yourself, WebCrawlerAPI handles scheduling, retries, and JS rendering for large crawls through a single API call.