What is crawl budget?

Crawl budget is the number of pages you can realistically fetch from a site in a given window — bounded by the target's tolerance for load and by your own infrastructure. On a small site, budget rarely matters. On a site with hundreds of thousands of pages, spending it on the wrong pages means the ones that matter never get crawled at all.

Common mistake

Crawling breadth-first with no prioritization, so budget gets consumed by whatever the crawler happens to find first:

def crawl(seed, budget=5000):
    queue, visited, fetched = [seed], set(), 0
    while queue and fetched < budget:
        url = queue.pop(0)
        if url in visited:
            continue
        visited.add(url)
        html = fetch(url)
        fetched += 1
        for link in extract_links(html):
            queue.append(link)  # every link enqueued with equal priority
    return visited

If the seed page links heavily into an infinite calendar widget or a faceted search with thousands of filter combinations, the budget gets consumed by those low-value pages before the crawler ever reaches the actual product catalog a few links deeper.

The fix

Score pages before crawling them and prioritize the queue by that score, so budget goes to what matters first:

import heapq
from urllib.parse import urlparse

def score(url: str) -> int:
    path = urlparse(url).path
    if "/product/" in path:
        return 10
    if "/category/" in path:
        return 5
    if "page=" in url or "?filter" in url:  # faceted search / pagination noise
        return -5
    return 1

def crawl_prioritized(seed, budget=5000):
    heap = [(-score(seed), seed)]
    visited, fetched = set(), 0

    while heap and fetched < budget:
        neg_score, url = heapq.heappop(heap)
        if url in visited:
            continue
        visited.add(url)
        html = fetch(url)
        fetched += 1
        for link in extract_links(html):
            if link not in visited:
                heapq.heappush(heap, (-score(link), link))

    return visited

Why it works

A priority queue keyed on a page-value score means high-value pages (product pages) get crawled before low-value ones (faceted filter combinations) regardless of which was discovered first in the link graph, so a limited budget is spent where it has the most impact. Negatively scoring known low-value patterns (pagination params, filter query strings) actively deprioritizes the pages most likely to explode the queue with near-duplicate variants of the same content.

Tips

• Faceted navigation and calendar-style infinite date ranges are the most common budget sinks — identify and deprioritize their URL patterns explicitly rather than discovering the problem after budget runs out.
• Recompute scores as you learn more — a category page that turns out to link to 200 products is worth more than your initial static score assumed.
• Crawl budget and crawl frequency are related but different levers — see how often you should crawl a site for the freshness side of this tradeoff.
• A tight budget is also a defense against getting blocked — see avoiding blocks when crawling for how pacing and budget interact.

WebCrawlerAPI applies sensible crawl prioritization automatically, so large sites don't require you to hand-tune a scoring function before your budget gets wasted on low-value pages.