How often should you crawl a site?

Crawl frequency is a tradeoff between data freshness and load — on the target site and on your own infrastructure. A fixed schedule applied uniformly across all pages wastes budget on content that never changes and misses updates on content that changes constantly.

Common mistake

Crawling every page on the same fixed interval regardless of how often it actually changes:

import schedule

def crawl_all_pages():
    for url in all_known_urls:  # includes the homepage and a 3-year-old archive page alike
        fetch(url)

schedule.every(1).hours.do(crawl_all_pages)

An hourly crawl of a static "About Us" page that hasn't changed in years wastes requests and load. Meanwhile a pricing page that updates daily might need more frequent checks than an hourly crawl provides if the business decision depends on same-day pricing data.

The fix

Track how often each page actually changes, and schedule per-page intervals based on observed change frequency, not a global default:

import hashlib
from datetime import datetime, timedelta

class PageSchedule:
    def __init__(self):
        self.state = {}  # url -> {content_hash, last_crawled, interval_hours}

    def should_crawl(self, url) -> bool:
        record = self.state.get(url)
        if not record:
            return True
        next_due = record["last_crawled"] + timedelta(hours=record["interval_hours"])
        return datetime.now() >= next_due

    def record_result(self, url, content):
        content_hash = hashlib.sha256(content.encode()).hexdigest()
        record = self.state.get(url, {"content_hash": None, "interval_hours": 24})

        if record["content_hash"] == content_hash:
            record["interval_hours"] = min(record["interval_hours"] * 2, 24 * 7)  # unchanged: back off
        else:
            record["interval_hours"] = max(record["interval_hours"] / 2, 1)  # changed: check more often

        record.update(content_hash=content_hash, last_crawled=datetime.now())
        self.state[url] = record

scheduler = PageSchedule()
for url in all_known_urls:
    if scheduler.should_crawl(url):
        content = fetch(url)
        scheduler.record_result(url, content)

Why it works

Content hashing turns "did this page change" into a cheap, objective check instead of a guess, and the adaptive interval — doubling when unchanged, halving when it changes — converges each page toward the frequency that actually matches its update rate: static pages drift toward weekly checks, volatile pages toward hourly ones, without hand-tuning a schedule per URL.

Tips

• Start conservative (once a day) for pages you have no change-history on, and let the adaptive logic tighten the interval once you have data.
• Watch for rate-limit responses as a signal too — if 429s appear at your current frequency, that's the site telling you to back off regardless of what your freshness goal says; see how to avoid getting blocked when crawling.
• Your own crawl budget is the other constraint — see what crawl budget is for how to prioritize which pages get the tighter intervals when you can't check everything as often as you'd like.

WebCrawlerAPI supports scheduled recurring crawls, so tracking freshness and re-fetching changed pages doesn't require building this scheduling logic yourself.