How do you avoid getting blocked when crawling?

Blocking at the crawl level is usually a pattern problem, not a per-request problem: a crawler that hits hundreds of pages on the same domain in a tight window looks nothing like organic traffic, regardless of how clean any individual request is. Avoiding it means shaping the crawl's overall traffic pattern, not just fixing headers.

Common mistake

Crawling as fast as the network allows, with concurrency bounded only by hardware:

import asyncio
import aiohttp

async def crawl_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [session.get(url) for url in urls]  # hundreds of concurrent requests
        return await asyncio.gather(*tasks)

A hundred simultaneous requests to one domain is a traffic spike no legitimate browsing session produces. Even if every individual request has a proper User-Agent, this pattern is exactly what rate-limiting and anti-bot systems are built to catch.

The fix

Bound concurrency per domain, respect any published Crawl-delay, and distribute requests over time instead of bursting them:

import asyncio
import aiohttp
from urllib.robotparser import RobotFileParser

async def crawl_politely(urls, seed_url, max_concurrent=2):
    rp = RobotFileParser()
    rp.set_url(f"{seed_url}/robots.txt")
    rp.read()
    delay = rp.crawl_delay("*") or 1.0

    semaphore = asyncio.Semaphore(max_concurrent)
    results = []

    async def fetch(session, url):
        async with semaphore:
            if not rp.can_fetch("*", url):
                return None
            resp = await session.get(url)
            await asyncio.sleep(delay)
            return resp

    async with aiohttp.ClientSession() as session:
        for url in urls:
            results.append(await fetch(session, url))
    return results

For a crawl spanning many domains, cap concurrency per domain, not globally — a global cap lets one large domain starve the others, while a per-domain cap keeps traffic to each site low regardless of overall job size.

Why it works

A per-domain concurrency limit (via Semaphore) keeps the burst size to a target site low no matter how large the overall crawl job is, since the limit is scoped to that one domain's traffic, not the crawler's total throughput. Reading and respecting Crawl-delay from robots.txt means the pacing matches what the site owner explicitly asked for, rather than a guessed interval that might still be too aggressive for that particular server.

Tips

• Distribute request timing with jitter, not a fixed interval — a crawler hitting a domain exactly every 2.000 seconds is as detectable as one with no delay at all.
• Watch response codes as a live signal: back off further on any 429/503, don't wait for a hard block to react.
• This is the crawl-strategy angle — pacing and concurrency across many pages. Per-request signals like headers and User-Agent are the scraping-side concern; see avoiding blocks when scraping.
• robots.txt compliance is table stakes here — see what robots.txt does if you haven't implemented that check yet.

WebCrawlerAPI manages per-domain pacing, concurrency, and robots.txt compliance internally, so crawling a large site doesn't require tuning this yourself.