Web Scraping in R: Beyond rvest — When to Use an API Instead of Code

R has a reputation as a stats language, not a scraping language. That's mostly wrong. rvest, part of the tidyverse, is a genuinely good tool for pulling data out of static HTML — it reads like a dplyr pipeline and gets you from URL to data frame in a handful of lines.

Where R scraping breaks down is not the parsing. It's everything around the parsing: pages that render with JavaScript, sites that block you after 20 requests, and the point where a "quick script" turns into something that needs to run reliably every day. I'll show the working code for both the easy case (rvest) and the harder case (JS-rendered pages), then talk about where hand-rolled R code stops being the right tool.

Scraping Static HTML with rvest

If the data you want is in the page's initial HTML — check with "View Page Source" in your browser, not just the rendered page — rvest is the right tool. No browser, no Selenium, no Docker. Just an HTTP request and a CSS selector.

Install it

install.packages("rvest")

rvest wraps xml2 for parsing and gives you a pipe-friendly API (|> or %>%) for selecting elements. If you already use the tidyverse, this will feel familiar.

Scrape a single page

This pulls book titles and prices off books.toscrape.com, a site built for scraping practice:

# install.packages("rvest")
library(rvest)

page <- read_html("https://books.toscrape.com/")

titles <- page |> html_elements(".product_pod h3 a") |> html_attr("title")
prices <- page |> html_elements(".product_pod .price_color") |> html_text2()

books <- data.frame(title = titles, price = prices)
print(head(books, 5))

What's happening here:

• read_html() fetches the page and parses it in one call. No separate request step.
• html_elements() takes a CSS selector and returns every matching node. .product_pod h3 a targets the link inside each book card's title.
• html_attr("title") reads an attribute — the book title is stored in the title attribute here, not the link text.
• html_text2() extracts visible text and normalizes whitespace (better than the older html_text() for real-world HTML with stray line breaks).

This is copy-paste runnable. Save it, run it, you get a data frame back.

Crawl multiple pages

Real scraping jobs rarely stop at one page. Here's the same selector logic looped across paginated catalogue pages:

# install.packages("rvest")
library(rvest)

base_url <- "https://books.toscrape.com/catalogue/page-%d.html"
all_books <- data.frame()

for (page_num in 1:3) {
  page <- read_html(sprintf(base_url, page_num))
  titles <- page |> html_elements(".product_pod h3 a") |> html_attr("title")
  prices <- page |> html_elements(".product_pod .price_color") |> html_text2()

  all_books <- rbind(all_books, data.frame(title = titles, price = prices))
  Sys.sleep(1) # be polite, avoid hammering the server
}

cat(nrow(all_books), "books scraped across 3 pages\n")
print(head(all_books, 5))

Key parts:

• sprintf(base_url, page_num) builds the paginated URL — swap this for whatever pattern the target site uses (?page=N, /page/N/, etc.).
• rbind() stacks each page's data frame onto the running result. Fine for a handful of pages; for hundreds, preallocate a list and bind_rows() from dplyr instead — repeated rbind() in a loop gets slow.
• Sys.sleep(1) is not decoration. Hit a site with no delay between requests and you'll get rate-limited or IP-banned fast. One second is a reasonable default; check the site's robots.txt for a Crawl-delay if it specifies one.

This will get you through most static-site scraping tasks: product listings, articles, directories, tables. The moment it breaks is JavaScript-rendered content. read_html() only sees what's in the initial HTML response — if the data loads in after the page renders, you get an empty selector match and no error telling you why.

Scraping JavaScript-Heavy Sites in R

Check first: open the target page, right-click, "View Page Source," and search for the text you're trying to scrape. If it's not there but it's visible on the normal page, the content is rendered client-side and rvest alone won't reach it.

The classic answer: RSelenium

RSelenium is the long-standing way to drive a real browser from R. It works, but it comes with real infrastructure overhead most people underestimate going in: you need a Selenium server running separately — either a standalone .jar process or a Docker container (docker run -d -p 4445:4444 selenium/standalone-chrome) — and RSelenium talks to it over HTTP. That's a second process to manage, keep alive, and version-match against your browser driver. For a one-off scrape, that's a lot of moving parts.

A lighter alternative: chromote

chromote drives a local headless Chrome instance directly over the Chrome DevTools Protocol — no Selenium server, no Docker, no separate process to babysit. It's an RStudio package built for exactly this kind of task. Here's a working example that scrapes a JS-rendered version of quotes.toscrape.com:

# install.packages(c("chromote", "rvest"))
# Needs a local Chrome/Chromium install, no Selenium server required.
library(chromote)
library(rvest)

b <- ChromoteSession$new()
invisible(b$Page$navigate("https://quotes.toscrape.com/js/", wait_ = TRUE))
Sys.sleep(1) # let the JS render the quotes

html <- b$Runtime$evaluate("document.documentElement.outerHTML")$result$value
invisible(b$close())

page <- read_html(html)
quotes <- page |> html_elements(".quote .text") |> html_text2()
print(head(quotes, 3))

What's happening here:

• ChromoteSession$new() launches (or connects to) a local headless Chrome instance. You need Chrome or Chromium installed, but nothing else running.
• Page$navigate(..., wait_ = TRUE) loads the URL and waits for the navigation to complete.
• Sys.sleep(1) gives the page's JavaScript time to actually populate the DOM after navigation finishes — navigation completing and content rendering are not the same event. For anything less predictable than a demo site, wait for a specific selector to appear instead of a fixed sleep.
• Runtime$evaluate("document.documentElement.outerHTML") pulls the fully-rendered HTML straight out of the browser's DOM, after JavaScript has run.
• The rendered HTML then goes right back into read_html() and the same rvest selector code you already know. chromote handles rendering, rvest handles parsing — you don't need to learn two different extraction APIs.

For a single JS-rendered page or a handful of them, chromote is the lower-friction choice over RSelenium. If you're already running a Selenium grid for other reasons, or need Selenium-specific features (multi-browser support, Selenium Grid distribution), stick with RSelenium. Otherwise, chromote gets you the rendered DOM with far less setup.

Where Both Approaches Hit a Wall at Scale

Both rvest and chromote work well for what they're built for: a script that runs on your machine, hits a reasonable number of pages, and finishes. Push past that and you start running into problems that have nothing to do with R and everything to do with running a scraper in production:

• IP bans and rate limits. Sites notice a single IP hitting hundreds of pages and start returning 403s or CAPTCHAs. Fixing this means rotating proxies — which means renting proxy infrastructure and writing retry logic around failed requests.
• Anti-bot defenses. Cloudflare challenges, fingerprinting, behavioral detection. A vanilla headless Chrome session (via chromote or Selenium) looks like a bot to a lot of modern anti-bot systems, even with real JavaScript execution.
• Headless browser infra. Chrome processes leak memory, crash under load, and need to be restarted and monitored if you're running this as a scheduled job rather than a one-off script. That's ops work, not R work.
• Retry and backoff logic. Networks fail, servers time out, pages render slowly. A script that works today will silently produce empty results tomorrow unless you build in retries with backoff — and then handle the case where retries also fail.
• Scheduling and state. If this needs to run daily against thousands of URLs, you now need a job queue, a way to track what succeeded and what didn't, and somewhere to store results.

None of this is an R problem specifically — you'd hit the same wall in Python or Node. It's the gap between "a scraping script" and "a scraping pipeline you can depend on."

When to Use an API Instead of R Code

The honest answer depends on what the job actually is:

• One-off script or research task — pulling a dataset for an analysis, checking a few hundred pages once, a personal project. rvest (or chromote if it needs JS) is the right call. Simple, free, no external dependency.
• Production data pipeline — something that needs to run on a schedule, hit a lot of pages, survive site changes, and not silently break when a target site adds bot detection. This is where proxy management, retries, and headless browser infrastructure stop being a one-time cost and start being ongoing maintenance. At that point, an API that already handles rendering, proxies, and anti-bot defenses is often less total work than maintaining that infrastructure yourself — even accounting for the cost.

If you're in the second bucket, R can still be your language of choice — you just don't write the crawling infrastructure yourself. Here's a working example using WebCrawlerAPI from R via httr2:

# install.packages(c("httr2", "jsonlite"))
# Get your key at https://dash.webcrawlerapi.com/access
library(httr2)
library(jsonlite)

api_key <- Sys.getenv("WEBCRAWLERAPI_API_KEY")
target_url <- "https://books.toscrape.com/"

# 1. Start the job
start_resp <- request("https://api.webcrawlerapi.com/v1/crawl") |>
  req_auth_bearer_token(api_key) |>
  req_body_json(list(url = target_url, items_limit = 1, scrape_type = "markdown")) |>
  req_perform() |>
  resp_body_json()

job_id <- start_resp$id
cat("Job started:", job_id, "\n")

# 2. Poll until the job is done
job_url <- paste0("https://api.webcrawlerapi.com/v1/job/", job_id)
repeat {
  job <- request(job_url) |>
    req_auth_bearer_token(api_key) |>
    req_perform() |>
    resp_body_json()

  if (job$status %in% c("done", "error")) break
  Sys.sleep(3)
}

# 3. Fetch the markdown content of the first crawled page
item <- job$job_items[[1]]
markdown <- request(item$markdown_content_url) |>
  req_perform() |>
  resp_body_string()

cat(substr(markdown, 1, 300), "\n")

What's happening here:

• req_auth_bearer_token() attaches your API key. You'll need a real key from dash.webcrawlerapi.com/access — without one, the request correctly returns a 401.
• The first POST starts a crawl job and returns immediately with a job ID — it doesn't block while pages are fetched, rendered, and processed on the other end.
• The repeat loop polls the job status every 3 seconds until it's done or error. This is the same pattern you'd use for any async job: start it, poll it, get results.
• job$job_items holds one entry per crawled page, each with a markdown_content_url — a link to fetch the actual page content in the format you asked for (markdown here; text and raw HTML are also options).

The proxy rotation, JavaScript rendering, retries, and anti-bot handling all happen on the API side. Your R code just starts a job, waits, and reads the result.

One-Off Script vs. Production Pipeline: Which to Use

Summary

rvest is a solid, low-friction tool for scraping static HTML in R — the code above is copy-paste runnable and will handle a large share of real scraping tasks. When a site renders content with JavaScript, chromote gets you the rendered DOM without standing up a Selenium server. Both approaches are the right call for one-off scripts and small research tasks.

Where they stop being the right call is scale: proxy rotation, anti-bot defenses, headless browser upkeep, and retry logic are infrastructure problems, not R problems, and they show up regardless of language. If the job is a recurring pipeline rather than a script you run once, an API like WebCrawlerAPI removes that infrastructure burden — you keep writing R, you just stop maintaining the scraping plumbing underneath it.