Is Web Scraping Legal? Court Cases, ToS, robots.txt and GDPR

Last updated October 2026. This is not legal advice.

Is web scraping legal? In the US, scraping public pages is generally not a crime and not "hacking". The risk comes from four places: logins and access controls, website terms (contract), copyright, and personal data.

So "is web scraping illegal" has no single answer. Data scraping is a method. What you scrape, how you get it, and what you do with it decide the outcome.

I'm Andrew, and I run a crawling API, so I read these cases more often than I'd like. Below you'll find the main rulings with dates, a risk table by data type, and a "can I scrape this?" checklist. If you want the case stories first, start with scrapers who won and scrapers who lost.

The short answer: when scraping is legal and when it is not

Here's the quick version. Every row is explained later in the post.

What you do	Usual risk	Why
Scrape public pages, no login, collect facts	Low	US courts have not treated this as unauthorized access
Scrape behind a login or paywall	High	Access rules and the site's terms apply
Collect personal data (names, emails, profiles)	Medium to high	Privacy law applies even if the data is public
Copy and republish articles, images, descriptions	Medium to high	Copyright covers expression, not facts
Keep going after a cease-and-desist or a block	High	Courts treat this as the moment access becomes "unauthorized"

Legality comes down to three things: what you take, how you get it, and what you do with it afterwards.

Is web crawling legal too?

Yes, under the same rules. Courts look at access (how you got in) and use (what you did with the data). They don't care much whether you call it crawling or scraping.

Search engines crawl the whole web under the same ideas. The practical difference is volume and intent, not the law. If the two terms are mixed up in your head, this post explains the difference.

Web scraping laws by region: US, EU and UK

There's no "web scraping law". You get a pile of general laws that happen to apply.

Region	Laws that usually matter
US	Computer Fraud and Abuse Act (CFAA), state computer-access laws, copyright, contract law (terms of service), CCPA/CPRA for personal data
EU	GDPR, the database right, the text and data mining opt-out in the Copyright Directive (Art. 4), the AI Act for AI training
UK	UK GDPR, UK copyright law

Other countries differ, and I won't make claims about them here. If you scrape from or about a specific country, ask a local lawyer.

US court cases that shape the answer

This is a summary with dates. Each ruling decided a narrow question, so I also note what it did not decide.

Public pages and the CFAA: Van Buren and hiQ v. LinkedIn

The CFAA is the US anti-hacking law. For years, companies argued that scraping "exceeds authorized access".

In Van Buren v. United States (Supreme Court, June 3, 2021), the Court read the law as a "gates-up-or-down" test. You are either allowed into a part of a system or you're not.

Then came hiQ Labs v. LinkedIn (9th Circuit, April 18, 2022). The court held that the CFAA concept of "without authorization" does not apply to public websites. hiQ scraped public LinkedIn profiles, and that was not "hacking".

What it did not decide: this was a preliminary injunction ruling, not a final verdict. And the story didn't end well for the scraper. In November 2022 the district court found that hiQ had breached LinkedIn's User Agreement, and the case ended in a settlement the following month (press reports mention a permanent injunction against hiQ; I haven't seen the full terms). Contract was the claim that stuck, which is the next section.

Terms of service and logged-off scraping: Meta v. Bright Data and X Corp v. Bright Data

In January 2024, Judge Chen sided with Bright Data on Meta's terms-of-service claim for scraping done while logged off. Meta dropped the case shortly after (per press reports). In May 2024, Judge Alsup dismissed X Corp's claims against Bright Data.

The pattern: public pages scraped without an account were hard to attack through contract. Both rulings are district court rulings, so they don't bind other courts.

After a cease-and-desist: Facebook v. Power Ventures and Ryanair v. Booking

Facebook v. Power Ventures (9th Circuit, 2016) is the old reference. Power kept accessing Facebook after receiving a cease-and-desist letter and after IP blocks. The court held that access was then "without authorization".

Ryanair v. Booking.com is the newer version. A jury found against Booking.com in July 2024. In January 2025 the district court set the verdict aside because Ryanair hadn't proven the $5,000 in losses the CFAA requires. But the court also held that the CFAA can reach content any member of the public can get with a free account, once the owner has sent a cease-and-desist. Ryanair appealed to the Third Circuit in July 2025. I couldn't find an appellate ruling, so treat the district court's reasoning as a signal and not as settled law.

Take-away: a notice plus block evasion is where cases get ugly. Logged-off public scraping and "I kept going after they told me to stop" are very different situations.

AI agents: Amazon v. Perplexity

On August 4, 2026, the Ninth Circuit vacated a preliminary injunction Amazon had won against Perplexity's Comet browser agent. The court found Amazon was unlikely to show that Perplexity "accessed" Amazon's computers. On the record before the court, the user accessed Amazon with an AI tool's help.

What it did not decide: it covers the CFAA and California's computer-access statute, on the facts of that record. A footnote says the outcome does not impair Amazon's ability to regulate access through its private terms of service. So contract claims stay open.

Where to read more

This section is a summary, not a replacement for the full stories. The two case roundups (won and lost) go deeper on older cases.

robots.txt: is it legally binding?

No, not by itself. The robots.txt standard (RFC 9309) says it plainly: "These rules are not a form of access authorization."

So ignoring robots.txt is not a crime on its own. But I wouldn't call it harmless:

It's evidence of notice. If a dispute starts, "the site told us no, in writing, in a machine-readable file" is not a good look.
It's the ethical baseline of the web. Ignoring it is a fast way to get blocked.
In the EU, a machine-readable opt-out can matter for text and data mining under Art. 4 of the Copyright Directive.
Under GDPR, the EDPB's draft Guidelines 03/2026 on web scraping for generative AI say that a missing robots.txt, or not following it, is not consent. The draft also points toward excluding sources that signal opposition (robots.txt, CAPTCHAs). It is a draft, open for consultation until October 30, 2026.

Some sites put legal terms right in robots.txt. This is the top of the New York Times file, captured in October 2026:

Captured October 2026.

You can check any site yourself:

curl -s https://example.com/robots.txt

And if you build a crawler, check every URL before you request it. The format has enough edge cases that I'd use a parser library:

// Node 18+
// npm i robots-parser
// Idea: fetch /robots.txt once per origin and check every URL against it.

import robotsParser from "robots-parser";

const cache = new Map(); // origin -> parsed robots

export async function getRobots(origin) {
  if (cache.has(origin)) return cache.get(origin);
  const robotsUrl = new URL("/robots.txt", origin).toString();
  const res = await fetch(robotsUrl);
  const txt = res.ok ? await res.text() : "";
  const robots = robotsParser(robotsUrl, txt);
  cache.set(origin, robots);
  return robots;
}

export async function isAllowed(url, userAgent = "MyCrawler/1.0") {
  const robots = await getRobots(new URL(url).origin);
  return robots.isAllowed(url, userAgent) !== false;
}

// Usage
console.log(await isAllowed("https://example.com/blog/post"));
console.log(await isAllowed("https://example.com/private/page"));

Remember that this check is input for your crawler. It's not legal clearance. A page can be allowed by robots.txt and still be off limits by contract.

In WebCrawlerAPI, robots.txt checking is the respect_robots_txt option (off by default). When it's on and a URL is disallowed, you get the blocked_by_robots_txt error.

Terms of service: when do they bind you?

Clickwrap: you clicked "I agree". This is the easiest kind to enforce.
Browsewrap: a "Terms" link in the footer. Weaker, because nobody agreed to anything visible.
Logged-in use: you created an account and accepted the terms. The strongest case against you.

The rule I use: if you must log in or accept something to get the data, assume the ToS applies. The Bright Data rulings above show that logged-off scraping of public pages is much harder to attack through contract. They don't make it risk-free.

Two more things. ToS can change, so re-check them if you scrape at scale. And outside the US, contract and database rules differ; the lost cases post covers the Ryanair v. Flightbox story from Europe.

Copyright and database rights: what you can copy

A safe mental model:

Facts and raw numbers (prices, SKUs, availability) are usually fine.
How facts are written, selected and presented can be protected.

So scraping prices is very different from scraping and republishing product descriptions or blog posts. If your output looks like a copy of the original page, you're too close.

What I'd do instead:

Extract only the fields you need.
Store the source URL next to each record.
If you show text back to users, keep it short and link to the source, or use an API or license.

"It's behind a paywall, but I can technically fetch it" is not a defense. It's the opposite.

In the EU there's also a database right that protects substantial investment in a database. The scrapers won post has the BHB v. William Hill story. For fair use in the US, the usual pair is Perfect 10 (thumbnails, fine) and AP v. Meltwater (a substitute product, not fine).

If your goal is extracting main article text for search or an AI knowledge base, this post on Mozilla Readability covers the tooling.

Scraping for AI training and AI search

This is where the 2025-2026 rulings matter.

Bartz v. Anthropic (June 2025): training on lawfully acquired books was ruled fair use. A pirated library was not covered. A $1.5 billion settlement got final approval on July 20, 2026 (Reuters).
Thomson Reuters v. Ross (Third Circuit, September 29, 2026): the court held that Ross's use of Westlaw headnotes to train a competing legal search tool was not fair use, and affirmed. The court itself pointed out that Ross's AI was not generative, so the ruling doesn't answer the generative AI question.

What I take from both: where the content comes from and whether the output substitutes for the source decide the outcome. An AI knowledge base built on sources you own, license, or that a customer owns carries much lower risk than a general-purpose dataset scraped from competitors. These are copyright rulings. They say nothing about GDPR or contract.

Personal data stays personal data when it's public. Names, emails, phone numbers, photos and profiles all count.

Under GDPR you need:

a lawful basis, usually legitimate interest (Art. 6(1)(f)), which requires a balancing test;
transparency: Art. 14 requires telling people you collected their data, with an exception when it would take disproportionate effort (Art. 14(5)(b));
minimisation: only what you need.

The EDPB's draft Guidelines 03/2026 (adopted for consultation July 7, 2026) say consent is usually not workable for web scraping and push legitimate-interest balancing plus mitigations like source exclusion, filtering and deletion. They're a draft, and scoped to private companies training generative AI.

The best-known fine in this area: the Dutch DPA fined Clearview AI EUR 30.5 million (announced September 3, 2024) for scraping images to build a facial recognition database.

In California, CCPA/CPRA excludes "publicly available" information from "personal information", but the definition (Civil Code 1798.140) is narrow. It covers lawfully available government records, information a business has a reasonable basis to believe the consumer or widely distributed media lawfully made available to the general public, and information a consumer shared without restricting it to a specific audience. Biometric information collected without the consumer's knowledge is explicitly not "publicly available". Don't assume "public" means "free to use".

Practical rules:

Avoid personal data unless you truly need it.
If you need it, minimise, document the reason, secure it, and delete it later.
Treat forums, social media and review sites with care. People post publicly but still expect context.

Risk matrix: what you scrape vs what can go wrong

This table is my judgment from experience, not a legal rating. Low, Medium or High is the overall risk for a typical project.

Data type	Access / CFAA	ToS / contract	Copyright	Privacy	Overall
Product prices and availability	Low	Low to medium	Low	Low	Low
Public docs and blogs (facts, links)	Low	Low	Medium if republished	Low	Low to medium
News articles (full text)	Low	Medium	High if republished	Low	Medium to high
Public social profiles	Low	Medium	Low	High	High
Logged-in content	High	High	Medium	Medium	High
User reviews	Low	Medium	Medium	Medium	Medium
Court and public records	Low	Low	Low	Medium to high	Medium (see court records post)
Emails and phone numbers	Low	Medium	Low	High	High
Health, financial, student data	Medium	High	Low	Very high	Stop and ask a lawyer

Can I scrape this? A decision checklist

Run these 7 questions in order. Each one ends in Go, Slow down, or Stop and ask.

Is there an official API or license? Yes: use it.
Is login or payment needed? Yes: Stop and ask.
Does robots.txt or the ToS say no? Yes: Stop and ask, or use an API.
Is there personal data? Yes: follow the privacy path (lawful basis, minimise, delete).
Will you republish expressive content? Yes: Stop and ask.
Is your volume low and rate limited? No: Slow down.
Did you receive any notice or block? Yes: Stop and ask.

If all seven are clean, you're at Go. It doesn't guarantee anything legally, but it removes the obvious mistakes.

Scrape politely: rate limits, User-Agent and backoff

Most ethical problems in scraping are operational. Fifty requests per second to a small site is not research. It's a small DDoS.

Good defaults:

Concurrency per host: 1-3.
Delay between requests: 500ms-3000ms, with jitter.
Respect Retry-After on 429. Back off on 5xx.
Stop when errors keep rising.
Use an honest User-Agent with a contact (email or URL). Don't pretend to be Chrome.

A tiny Node 18+ pattern:

// Node 18+
// Idea: per-host delay + 429 backoff + honest user agent.

const nextAt = new Map(); // host -> unix ms

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function politeFetch(url, { minDelayMs = 800 } = {}) {
  const host = new URL(url).host;

  const waitMs = Math.max(0, (nextAt.get(host) ?? 0) - Date.now());
  if (waitMs) await sleep(waitMs);

  // Jitter, so you don't look like a metronome.
  const jitter = Math.floor(Math.random() * 400);
  nextAt.set(host, Date.now() + minDelayMs + jitter);

  const res = await fetch(url, {
    headers: { "user-agent": "MyCrawler/1.0 (+mailto:[email protected])" },
  });

  // The site says slow down: listen.
  if (res.status === 429) {
    const retryAfter = Number(res.headers.get("retry-after") ?? "0");
    nextAt.set(host, Date.now() + (retryAfter > 0 ? retryAfter * 1000 : 10_000));
  }

  return res;
}

If you can't afford to crawl slowly, you probably can't afford the problems that come with crawling fast. Building your own crawler? How to build a web crawler covers scheduling, and web scraping with JavaScript and Node.js covers the fetching side. For blocks and bot management, see what a Cloudflare web crawler is.

What to do if you get a cease-and-desist or a block

Don't panic, and don't get clever.

Don't ignore it, and don't evade it. Block evasion after notice is the pattern in the Power Ventures and Ryanair cases.
Pause the affected source while you figure it out.
Keep your logs: what you collected, when, and how.
Talk to a lawyer before you answer.
Ask for API access or a license. Often that's what the sender wants anyway.

When to ask permission, or use an API instead

If you scrape a few pages for a personal script, you won't email anyone. At scale, I'd ask when the ToS is strict or unclear, robots.txt blocks the paths you need, volume is high, or the data is sensitive.

Often the answer is "use our API" or "here's a dump". That's a win: faster, cheaper, and more stable. Scraping a bank or app screen with a customer's login is a riskier case of its own, covered in what is screen scraping.

Before scraping, also look at RSS feeds, sitemaps, bulk exports and public datasets. If you're comparing options, see what a web crawling API is and the best web scraping APIs.

If a site offers an API or allows crawling, a web scraping API for single pages or a crawling API for whole sites removes the headless browser and retry work. You still choose the URLs, and the legal responsibility for lawful use stays with you.

FAQ

Is web scraping legal?

Scraping public pages is generally legal in the US. Trouble comes from logins, terms of service, copyright and personal data. It's the combination of what you take, how, and what you do afterwards that counts.

Is web scraping illegal in the US? Can I go to jail for it?

Scraping public pages is not "hacking" under the CFAA after Van Buren (2021) and hiQ (2022). Criminal exposure comes with bypassing logins or access controls. Civil claims (contract, copyright) are the more common risk. Still, check with a lawyer for anything high-stakes.

Is it legal to scrape a website that says "no scraping" in its terms?

Breaking terms is a contract issue, not a crime. It matters most when you log in or accept the terms. Logged-off scraping of public pages was hard to attack by contract in the Bright Data cases, but those were district court rulings.

Do I have to follow robots.txt? Is robots.txt legally binding?

Not by itself: RFC 9309 says it is not access authorization. But it's evidence of notice, and in the EU it can act as a text-and-data-mining opt-out. I follow it.

Is web crawling legal?

Crawling falls under the same rules as scraping. Courts look at access and use. See crawling vs scraping.

Only with a lawful basis, usually legitimate interest, plus transparency and minimisation. Public data is still personal data. The EDPB's draft 2026 guidelines say consent is usually not workable for scraping.

Can I use scraped data to train AI or build an AI knowledge base?

It depends on the source and the output. Bartz (2025) treated training on lawfully acquired books as fair use, while Thomson Reuters v. Ross (2026) rejected fair use for a competing search tool. Use content you own or license when you can.

What if I receive a cease-and-desist letter?

Pause the affected source, keep logs, don't evade blocks, and talk to a lawyer. Ask about API access or a license.

Not legal advice

Laws and rulings change. This post reflects my review in October 2026 and my experience running a crawler, not counsel. For high-risk projects (personal data, logged-in content, republishing), talk to a lawyer.

Our own position is simple: customers are responsible for lawful use and for following target site terms and robots directives, and the service can't be used for unlawful activity or copyright infringement. See our terms and privacy policy.