What are ethical web scraping practices?

Ethical scraping is about what happens to the data after extraction, not just how politely you fetch it. A scraper can respect every rate limit and still cause harm — by collecting personal data without a basis, or republishing content that isn't yours to republish. Politeness and ethics are related but not the same thing.

Common mistake

Treating rate-limiting as the whole ethics checklist:

import time

for url in profile_urls:
    resp = requests.get(url, headers={"User-Agent": "MyBot/1.0 (contact@mydomain.com)"})
    time.sleep(1)  # "polite" pacing
    profile = extract_profile(resp.text)  # name, email, location, photo
    store_and_republish(profile)  # the actual ethical problem

The pacing and identifiable user agent are good practice, but they don't address the real issue: extracting personal data (emails, locations, photos) and republishing it without consent or a legal basis. Slowing down the request rate doesn't make that part fine.

The fix

Separate the two concerns — request behavior and data handling — and apply a check to each:

ALLOWED_FIELDS = {"product_name", "price", "availability"}  # explicit allowlist, no personal data

def extract_product(html):
    soup = BeautifulSoup(html, "html.parser")
    record = {
        "product_name": soup.select_one(".title").text.strip(),
        "price": parse_price(soup.select_one(".price").text),
        "availability": soup.select_one(".stock").text.strip(),
    }
    assert set(record.keys()) <= ALLOWED_FIELDS, "extraction pulled a field outside scope"
    return record

def scrape_politely(url):
    resp = requests.get(
        url,
        headers={"User-Agent": "MyCompanyBot/1.0 (+https://mycompany.com/bot; contact@mycompany.com)"},
    )
    time.sleep(1)
    return resp

Why it works

The ALLOWED_FIELDS assertion turns "don't collect personal data" from a policy someone remembers into a check the code enforces — if a future selector change accidentally pulls a .email field, the assertion fails loudly instead of silently shipping personal data into storage. An identifiable user agent with a contact URL lets a site owner reach you before escalating to a block, which is the actual courtesy politeness is meant to convey — not just the request pacing.

Tips

• Scope extraction to the minimum fields the use case needs. If you don't collect a field, you can't mishandle it later.
• Publish a real contact method in your user agent string — it's the difference between a site owner emailing you about unexpected load and silently blacklisting your IP range.
• Respect robots.txt disallow rules even where they aren't legally binding — they're the site owner's explicit statement of what they don't want crawled, and ignoring them is where "ethical" and "legal" diverge fastest.
• Ethics and legality aren't the same axis — see is web scraping legal for where contract and privacy law actually apply.

If you're scraping at a scale where politeness and data scoping need to be enforced consistently across many jobs, WebCrawlerAPI handles the request-level behavior for you, so your review effort can focus on what data you're extracting rather than how you're fetching it.