Is web scraping legal?

Scraping itself — an HTTP request to a public URL — isn't illegal. What creates legal exposure is what you do with the data you extract: republishing copyrighted text, harvesting personal information, or ignoring a site's terms of service after agreeing to them. The risk lives in the data and the use case, not the act of fetching a page.

Common mistake

Treating "the page is public" as a blanket legal clearance:

# Scrapes user profile data including names, emails, and photos
# then republishes it on a competing directory site
profiles = scrape_all_profiles("https://example-social-network.com")
publish_to_directory(profiles)

Public visibility doesn't waive copyright on the text and images, and it doesn't waive data-protection obligations (GDPR, CCPA) that attach the moment you collect personal data — regardless of whether the person "posted it publicly" themselves.

The fix

Scope the scrape to what you actually need and check the constraints before writing the pipeline, not after:

import requests
from urllib.robotparser import RobotFileParser

def is_allowed(url: str, user_agent: str = "*") -> bool:
    rp = RobotFileParser()
    rp.set_url("https://example.com/robots.txt")
    rp.read()
    return rp.can_fetch(user_agent, url)

if is_allowed("https://example.com/product/123"):
    # Extract only the fields the use case requires — price, name, stock —
    # not full profile pages or personal data that isn't needed
    data = scrape_product("https://example.com/product/123")

Then read the target site's terms of service for an explicit anti-scraping clause, and separately assess whether any field you collect counts as personal data under the laws your users or your company are subject to.

Why it works

Checking robots.txt and terms of service before scraping doesn't make an infringing use legal, but it does surface the two biggest real-world risk sources early: sites that have explicitly prohibited automated access (breach of contract risk) and fields that trigger privacy law (GDPR/CCPA risk). Narrowing the scrape to only the fields the use case needs also shrinks both risks — you can't leak personal data you never collected.

Tips

• Terms-of-service violations are a contract law question (can the site sue you for breach), not a criminal one — but courts have enforced them in scraping cases, so "public data" isn't a defense on its own.
• Scraping personal data (names, emails, faces in photos) triggers GDPR/CCPA obligations the moment you store it, independent of where you got it.
• Bypassing a login wall or paywall to reach data you scrape is a materially higher-risk category than scraping unauthenticated public pages.
• See what ethical scraping practices look like for the operational side of staying inside these lines.

When the stakes are real — high-volume commercial use, personal data, or a gray-area target — get legal review before you build the pipeline, not after. Read more in our Web Scraping Ethics: What is legal and what is not? post.