How do you crawl JavaScript-heavy sites?

On a single-page app, the links that lead to other pages often don't exist in the initial HTML at all — they're generated by client-side routing after JavaScript runs. A crawler that only reads the raw HTML response finds zero links and stops after the seed page, even though the site has thousands of routes.

Common mistake

Discovering links by parsing the raw HTTP response, which works for server-rendered sites but silently fails on SPAs:

def discover_links(url):
    html = requests.get(url).text
    soup = BeautifulSoup(html, "html.parser")
    return [urljoin(url, a["href"]) for a in soup.select("a[href]")]

links = discover_links("https://example.com/app")
# returns [] — the <div id="root"> is empty until React/Vue renders client-side

The crawl queue empties after the seed URL, and the crawler reports "site fully crawled" after visiting exactly one page — a false-complete result that's easy to miss unless you're checking output volume against expectations.

The fix

Render the page in a headless browser before extracting links, so client-side routing has already populated the DOM:

from playwright.sync_api import sync_playwright
from urllib.parse import urljoin, urlparse

def discover_links_rendered(url, domain):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        hrefs = page.eval_on_selector_all("a[href]", "els => els.map(e => e.href)")
        browser.close()
    return [h for h in hrefs if urlparse(h).netloc == domain]

def crawl_spa(seed_url, max_pages=500):
    domain = urlparse(seed_url).netloc
    queue, visited, pages = [seed_url], set(), []
    while queue and len(pages) < max_pages:
        url = queue.pop(0)
        if url in visited:
            continue
        visited.add(url)
        pages.append(url)
        queue += [u for u in discover_links_rendered(url, domain) if u not in visited]
    return pages

Many SPAs also expose a route manifest or sitemap even though content renders client-side — checking for that first avoids rendering-cost entirely for discovery.

Why it works

wait_until="networkidle" waits for the page's own network activity to settle before reading the DOM, giving client-side routing time to populate the anchor tags a plain HTTP request would never see. Filtering discovered links against domain keeps the crawl scoped correctly, same as with server-rendered sites — rendering changes how links are found, not the traversal logic around them.

Tips

• Check for sitemap.xml before committing to rendered crawling for discovery — many SPAs still publish one for search engines, which sidesteps the rendering cost entirely.
• Rendering every page during discovery is expensive at scale — cache the rendered link list per page and re-crawl only when you have reason to think routes changed.
• This is the discovery-side problem — finding which routes exist. Extracting data from each of those pages once found is a related but separate step; see scraping JavaScript-heavy sites.
• Keep concurrency low when rendering — a headless browser instance costs far more memory and CPU per page than a plain HTTP request, so the safe concurrency ceiling is much lower.

WebCrawlerAPI renders JavaScript-heavy sites during crawling automatically, so route discovery on SPAs works out of the box without you managing headless browser infrastructure.