Is web crawling legal?

Crawling's legal exposure lives mostly at the access-control layer: did you bypass something meant to keep you out. A crawler that fetches public pages a browser could reach is on very different legal footing than one that circumvents a login wall, ignores explicit access rules, or overwhelms a server hard enough to constitute interference.

Common mistake

Treating "the page loads without a login" as the only check needed:

def crawl(seed_url):
    # No robots.txt check, no rate limiting, crawls whatever is link-reachable
    return crawl_all_links(seed_url, max_pages=100_000)

crawl("https://example.com")

This ignores two separate risk sources: whether the site has published an explicit access policy (robots.txt, terms of service) and whether the request volume itself could be read as interference with the service — both matter independent of whether authentication was involved.

The fix

Check robots.txt before adding a URL to the crawl queue, and rate-limit to a level a human wouldn't consider abusive:

from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse
import time

def build_robot_parser(seed_url):
    parsed = urlparse(seed_url)
    rp = RobotFileParser()
    rp.set_url(f"{parsed.scheme}://{parsed.netloc}/robots.txt")
    rp.read()
    return rp

def crawl(seed_url, max_pages=1000, delay=1.0):
    rp = build_robot_parser(seed_url)
    queue, visited, pages = [seed_url], set(), []

    while queue and len(pages) < max_pages:
        url = queue.pop(0)
        if url in visited or not rp.can_fetch("*", url):
            continue
        visited.add(url)
        pages.append(requests.get(url))
        time.sleep(delay)
        # ... extend queue with discovered links ...

    return pages

Why it works

Checking robots.txt before fetching, rather than after, means the crawler never requests a path the site owner has explicitly excluded — the strongest signal courts and site owners both treat as evidence of good faith. A deliberate rate limit (delay) keeps request volume in a range that can't plausibly be argued as degrading the target's service, which is the other axis courts look at in crawling disputes, separate from what data was collected.

Tips

• robots.txt is a convention, not an access-control mechanism — respecting it is a strong good-faith signal, but ignoring it isn't automatically illegal on its own; it depends on jurisdiction and what happens next.
• Bypassing a login, CAPTCHA, or IP block to continue crawling is a materially different legal category than crawling openly accessible pages — this is where "authorized access" case law actually turns.
• Crawl volume that measurably degrades a target site's performance can be treated as interference independent of what data was collected — rate limiting isn't just etiquette.
• The data-usage side of this — copyright, personal data, ToS on what you scrape from crawled pages — is covered separately in is web scraping legal.

Read more in our Web Scraping Ethics: What is legal and what is not? post, and when the stakes are high — commercial scale, gray-area targets — get legal review before building the crawler.