How do you avoid getting blocked when scraping?

Blocking at the scraper level is usually triggered by the shape of individual requests, not overall traffic volume — a missing header, an identical fingerprint on every request, or a burst pattern no browser produces. Fixing it means making each request look like the one a real browser would send, not just slowing down.

Common mistake

Sending requests with library defaults and no variation:

import requests

for url in product_urls:
    resp = requests.get(url)  # default User-Agent: "python-requests/2.31.0"
    parse(resp.text)

A User-Agent string that announces the HTTP library, identical headers on every request, and no delay between them is a fingerprint sites detect within a handful of requests — well before you'd hit any documented rate limit.

The fix

Set realistic headers, add jitter between requests, and back off on retryable responses:

import requests
import random
import time

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

def scrape_with_backoff(url, max_retries=4):
    for attempt in range(max_retries):
        resp = requests.get(url, headers=HEADERS, timeout=10)
        if resp.status_code == 200:
            return resp
        if resp.status_code in (429, 503):
            retry_after = int(resp.headers.get("Retry-After", 2 ** attempt))
            time.sleep(retry_after)
            continue
        resp.raise_for_status()
    raise RuntimeError(f"Failed after {max_retries} retries: {url}")

for url in product_urls:
    resp = scrape_with_backoff(url)
    parse(resp.text)
    time.sleep(random.uniform(1, 3))  # jitter, not a fixed interval

Why it works

A realistic User-Agent and language header pass the cheapest layer of bot detection, which just checks whether the request looks like a browser at all. Randomized delays (jitter) avoid the metronomic timing pattern — requests exactly N seconds apart — that's a stronger signal of automation than raw request volume. Respecting Retry-After and backing off on 429/503 responses means the scraper cooperates with the rate limit the server is actually communicating, instead of retrying immediately and escalating into a harder block.

Tips

• Rotating IPs helps when the block is IP-based, but it doesn't fix a fingerprint problem — a botnet of requests with identical headers gets flagged just as fast from different IPs.
• Concurrency matters as much as per-request delay: 20 parallel requests with 2-second jitter each is still a burst pattern from the server's point of view.
• This is the extraction-request angle; if you're also crawling many pages to find URLs, pacing at the crawl-schedule level is a related but separate concern — see avoiding blocks when crawling.
• A 429 specifically means "you're going too fast," not "you're blocked" — see what a 429 error means for the distinction.

WebCrawlerAPI handles headers, rendering, and retry/backoff for you, so scraping sites with active bot detection doesn't require building this logic yourself.