How do you scrape JavaScript-heavy sites?

On a JavaScript-heavy site, the HTML your scraper fetches with a plain request and the DOM a visitor actually sees are different documents. Extraction has to happen after the browser has run the page's JavaScript, not before — otherwise you're parsing an empty shell.

Common mistake

Fetching the page like static HTML and finding the target field missing:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com/spa-product/123").text
soup = BeautifulSoup(html, "html.parser")
price = soup.select_one(".price")  # None — price is injected by JS after load

The response body contains a near-empty <div id="root"></div> and a bundle of JavaScript. The price genuinely isn't in that HTML; no amount of selector tuning fixes it.

The fix

Render the page in a headless browser and wait for the specific element to appear before extracting:

from playwright.sync_api import sync_playwright

def scrape_rendered_price(url: str) -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="domcontentloaded")
        page.wait_for_selector(".price", timeout=10_000)
        price = page.text_content(".price")
        browser.close()
        return price

price = scrape_rendered_price("https://example.com/spa-product/123")

If the price is fetched from a JSON API under the hood, intercepting that call is often faster and more reliable than waiting on the DOM:

from playwright.sync_api import sync_playwright

def scrape_via_api_intercept(url: str) -> dict:
    captured = {}
    with sync_playwright() as p:
        page = p.chromium.launch().new_page()
        page.on("response", lambda r: captured.update(r.json())
                if "/api/product/" in r.url and r.status == 200 else None)
        page.goto(url)
        page.wait_for_timeout(2000)
    return captured

Why it works

wait_for_selector blocks until the target element exists in the rendered DOM, so extraction only runs after the client-side JavaScript that builds it has finished — avoiding the race condition of reading the DOM too early. Intercepting the network response instead of the DOM sidesteps rendering entirely: if the front end already fetches structured JSON from an API, reading that response directly is both faster and less brittle than depending on a CSS selector that a redesign can break.

Tips

• Prefer wait_for_selector over a fixed sleep() — a fixed delay is either too short under load or wastes time when the page is already ready.
• Always check the browser's network tab for the underlying API call before committing to DOM scraping; it's usually faster and survives front-end redesigns better.
• Rendering is 10-50x more resource-intensive per page than a plain request — keep concurrency low and cache rendered results when the data doesn't change often.
• If you're also crawling many such pages to find URLs first, rendering to discover links is a related but distinct problem — see crawling JavaScript-heavy sites.

WebCrawlerAPI renders JavaScript-heavy pages for you and returns the final content as clean markdown or JSON, so you skip maintaining Playwright infrastructure and selector logic entirely.