How do you handle pagination when scraping?

Pagination is where scrapers silently lose data: a job that "works" but only returns page one, or a loop that never terminates because the "no more results" condition was never actually checked. Handling it correctly means detecting the end condition explicitly, not assuming it.

Common mistake

Looping a fixed number of times, or looping until a request fails:

results = []
for page in range(1, 100):  # arbitrary guess at how many pages exist
    resp = requests.get(f"https://example.com/products?page={page}")
    data = resp.json()
    results += data["items"]

If the site has 40 pages, this scrapes 60 empty or duplicate requests. If it has 150, you silently miss two-thirds of the data. Neither failure is visible unless someone checks the count.

The fix

Stop based on the actual signal the API or page gives you — an empty result set, a missing "next" link, or a cursor field — not a guessed page count:

def scrape_all_pages(base_url):
    results = []
    page = 1
    while True:
        resp = requests.get(base_url, params={"page": page})
        data = resp.json()
        items = data.get("items", [])
        if not items:
            break
        results += items
        if not data.get("has_next", False):
            break
        page += 1
    return results

For cursor-based APIs, the loop keys off the cursor instead of a page number, and resuming after a crash is a matter of storing the last cursor:

def scrape_with_cursor(base_url, resume_cursor=None):
    results, cursor = [], resume_cursor
    while True:
        resp = requests.get(base_url, params={"cursor": cursor} if cursor else {})
        data = resp.json()
        results += data["items"]
        cursor = data.get("next_cursor")
        if not cursor:
            break
        save_checkpoint(cursor)  # persist so a crash can resume, not restart
    return results

Why it works

Checking has_next or next_cursor ties the loop's termination to what the server actually reports, so it self-corrects if the page count changes between runs — no hardcoded upper bound to go stale. Persisting the cursor as a checkpoint means a failed job resumes from where it stopped instead of re-scraping pages you already have, which matters once a "page" job takes long enough to realistically fail partway through.

Tips

• Always keep a hard iteration cap (while page < 10_000) even with a correct end condition — it protects against an API bug that never sets has_next to false.
• For infinite-scroll pages with no paginated API underneath, you need a rendered browser to trigger the scroll and capture new DOM nodes as they load, not a plain HTTP loop.
• Log the total item count per run and alert on large drops — a silent pagination break shows up first as "why did we get 90% fewer records today."

If the target site doesn't expose a clean paginated API, WebCrawlerAPI handles scroll-triggered and JS-rendered pagination for you and returns the full result set from one job.