How do you crawl JavaScript-heavy sites?

On a single-page app, the links that lead to other pages often don't exist in the initial HTML at all — they're generated by client-side routing after JavaScript runs. A crawler that only reads the raw HTML response finds zero links and stops after the seed page, even though the site has thousands of routes. For a Node.js version, see the JavaScript web crawler page.

Common mistake

Discovering links by parsing the raw HTTP response, which works for server-rendered sites but silently fails on SPAs:

def discover_links(url):
    html = requests.get(url).text
    soup = BeautifulSoup(html, "html.parser")
    return [urljoin(url, a["href"]) for a in soup.select("a[href]")]

links = discover_links("https://example.com/app")
# returns [] — the <div id="root"> is empty until React/Vue renders client-side

The crawl queue empties after the seed URL, and the crawler reports "site fully crawled" after visiting exactly one page — a false-complete result that's easy to miss unless you're checking output volume against expectations.

The fix

Render the page in a headless browser before extracting links, so client-side routing has already populated the DOM:

from playwright.sync_api import sync_playwright
from urllib.parse import urljoin, urlparse

def discover_links_rendered(url, domain):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        hrefs = page.eval_on_selector_all("a[href]", "els => els.map(e => e.href)")
        browser.close()
    return [h for h in hrefs if urlparse(h).netloc == domain]

def crawl_spa(seed_url, max_pages=500):
    domain = urlparse(seed_url).netloc
    queue, visited, pages = [seed_url], set(), []
    while queue and len(pages) < max_pages:
        url = queue.pop(0)
        if url in visited:
            continue
        visited.add(url)
        pages.append(url)
        queue += [u for u in discover_links_rendered(url, domain) if u not in visited]
    return pages

Many SPAs also expose a route manifest or sitemap even though content renders client-side — checking for that first avoids rendering-cost entirely for discovery.

Why it works

wait_until="networkidle" waits for the page's own network activity to settle before reading the DOM, giving client-side routing time to populate the anchor tags a plain HTTP request would never see. Filtering discovered links against domain keeps the crawl scoped correctly, same as with server-rendered sites — rendering changes how links are found, not the traversal logic around them.

Tips

• Check for sitemap.xml before committing to rendered crawling for discovery — many SPAs still publish one for search engines, which sidesteps the rendering cost entirely.
• Rendering every page during discovery is expensive at scale — cache the rendered link list per page and re-crawl only when you have reason to think routes changed.
• This is the discovery-side problem — finding which routes exist. Extracting data from each of those pages once found is a related but separate step; see scraping JavaScript-heavy sites.
• Keep concurrency low when rendering — a headless browser instance costs far more memory and CPU per page than a plain HTTP request, so the safe concurrency ceiling is much lower.

WebCrawlerAPI renders JavaScript-heavy sites during crawling automatically, so route discovery on SPAs works out of the box without you managing headless browser infrastructure.