What is a 429 error in web scraping?

A 429 is the server telling your scraper "too many requests, slow down" — not "you're banned." It's a rate-limit signal, distinct from a 403 (forbidden) or a hard IP block. Treating it as the same failure as a ban means either giving up too early or retrying too fast, both of which make the problem worse.

Common mistake

Retrying immediately, or treating a 429 the same as any other error and aborting:

resp = requests.get(url)
if resp.status_code == 429:
    resp = requests.get(url)  # immediate retry — the server is still rate-limited
elif resp.status_code != 200:
    raise Exception(f"Failed: {resp.status_code}")

Retrying immediately on a 429 doesn't help — the rate window the server is enforcing hasn't reset, so the second request fails the same way, often escalating into a longer block if the server tracks repeated violations.

The fix

Read the Retry-After header if present, and fall back to exponential backoff if it isn't:

import time
import requests

def scrape_with_retry(url, max_retries=5):
    for attempt in range(max_retries):
        resp = requests.get(url)
        if resp.status_code == 200:
            return resp
        if resp.status_code == 429:
            wait = resp.headers.get("Retry-After")
            wait = int(wait) if wait else 2 ** attempt  # exponential fallback
            time.sleep(wait)
            continue
        resp.raise_for_status()
    raise RuntimeError(f"Still rate-limited after {max_retries} retries: {url}")

If 429s are frequent rather than occasional, the real fix is upstream of retry logic — lower your steady-state request rate:

import time

REQUESTS_PER_SECOND = 0.5  # one request every 2 seconds, sustained

def scrape_all(urls):
    results = []
    for url in urls:
        results.append(scrape_with_retry(url))
        time.sleep(1 / REQUESTS_PER_SECOND)
    return results

Why it works

Retry-After is the server explicitly stating how long to wait, so honoring it resolves the block on the first retry instead of guessing. Exponential backoff as a fallback avoids hammering a server that doesn't send the header, spacing retries out further each time until they succeed. Lowering the steady-state rate addresses the actual cause — if you're hitting 429s regularly, retry logic is compensating for a request rate the target site has already told you is too high.

Tips

• A 429 with no Retry-After header still means "slow down" — default to a conservative backoff (start at 2-4 seconds) rather than retrying fast.
• If retries consistently fail even with backoff, you've likely escalated to a 403 or an IP block — a different failure mode requiring different handling, not more waiting.
• Concurrent requests count against the same rate limit as sequential ones — throttle total concurrency, not just per-worker delay, if you're scraping with multiple threads or processes.
• This is one instance of the broader blocking problem — see how to avoid getting blocked when scraping for the full picture.

WebCrawlerAPI handles rate-limit detection and backoff internally, so a 429 from the target site is retried correctly without you writing this logic yourself.