Best Web Crawler API in 2026

Full disclosure before anything else: I run WebCrawlerAPI, which is first on this list. I've tried to be fair about where we lose. If you work on one of the other products here and I got something wrong or out of date, email me at hello@webcrawlerapi.com and I'll fix it.

Picking a crawler API is not a small decision. Whatever you choose gets wired into your ingestion pipeline, your billing forecast, and your on-call rotation. Swapping it out six months later means rewriting the part of your system you understand least.

So this post compares the five products people usually shortlist for this job - on the things that actually decide it: can it crawl a whole site from one seed URL, does it render JavaScript, what comes out the other end, and what it costs per 1,000 pages. Two spoilers, because you'd find out anyway: one of the five shut down, and one of them isn't really a crawler. If you're still fuzzy on the difference between crawling and scraping, start here. If you only ever need one page at a time, you want the web scraping API comparison instead.

I re-checked every price and feature below against each vendor's own live pages in September 2026. Two of the five changed enough since the last version of this post that the old writeups were simply wrong. Vendor pricing moves fast - verify before you commit budget.

Quick comparison

Two cells need footnotes.

Apify doesn't price per page, and no honest conversion exists. A compute unit is 1 GB of RAM running for one hour. How many pages that buys depends on the actor, the site, and whether the browser is rendering. Anyone publishing a clean "Apify costs $X per 1,000 pages" number made it up.

Crawlbase's ladder is per successful request, and the headline rates are for standard pages. JS rendering and site difficulty apply multipliers whose values aren't on the pricing page. So the $0.76 blended rate their calculator shows for 100K requests is a floor, not a forecast.

1. WebCrawlerAPI

I built this one, so read the pros with appropriate suspicion and the cons carefully.

The job it's designed for is narrow: you have a domain - docs site, help center, marketing site - and you want every page of it as clean Markdown, indexed into an AI knowledge base or search product. You don't want to run Playwright, rotate proxies, or write yet another boilerplate stripper.

You give it a seed URL and a page limit. It maps the site, follows links, renders JavaScript, and returns Markdown, cleaned text, raw HTML, or the link graph. Menus, footers, cookie banners, and ads get stripped before you see anything.

// Node 18+. Kick off a crawl and get the job id back.
const res = await fetch("https://api.webcrawlerapi.com/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.WEBCRAWLER_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://docs.stripe.com",
    items_limit: 200,          // hard stop, so a big site can't drain your balance
    output_formats: ["markdown"],
    max_depth: 2,              // 0 = seed page only
    respect_robots_txt: true,
  }),
});

const { id } = await res.json();  // crawling is async - poll the job or use webhook_url
console.log(id);

Full parameter list is in the crawl endpoint docs. The controls that matter in practice are max_depth, whitelist_regexp, and blacklist_regexp - you'll want them the first time a crawl wanders into a site's paginated tag archive.

Two other pieces worth knowing. Feeds handle re-crawling: point a feed at a site and get back only the pages that changed, with diffs, instead of re-ingesting everything weekly. And the Webcrawler Agent takes a plain-English prompt plus a seed URL, browses across pages, and returns structured JSON - useful when what you want isn't "the content" but "every pricing tier listed anywhere on this site."

Pros

• Seed URL in, clean Markdown out, with depth and regex controls that behave predictably
• Four output formats including cleaned text and the raw link graph
• Pay-as-you-go with no monthly fee - you can use it at $0/month forever
• JS rendering, retries, and proxy rotation are included, not add-ons
• Feeds for incremental re-crawls
• SDKs for JavaScript/TypeScript, Python, PHP, and .NET

Cons

• No authenticated crawling. No cookie jar, no custom headers, no session handling. Anything behind a login is out of reach. Crawlbase and Apify both beat us here.
• No sitemap-first mode. Crawling is link-discovery only. If a site's sitemap is better than its internal linking, you can't take advantage of it.
• We get expensive at volume. More on this in a second.

Pricing

Pay-as-you-go starts at $2 per 1,000 pages with no monthly fee and no card required to sign up. The $29/month Starter tier is the same $2 per 1,000 - what it buys is concurrency, 10 parallel requests instead of 5. The higher tiers are where the rate actually drops: $99/month takes it to $1.50 per 1,000 and 20 parallel requests, $499/month to $1 and 50. A handful of hard sites - TikTok, LinkedIn public pages - are priced above the standard rate.

That's fine at 50K pages a month. It is not fine at 10 million. Crawlbase's published ladder works out around $0.15 per 1,000 at that volume - roughly one seventh of our best rate. If you crawl at that scale and the pages are mostly simple, we're the wrong choice on price, and I'd rather say so here than have you find out on an invoice.

Where I'd still argue we win is everything below that line: zero to clean Markdown for a whole domain, no card, no subscription, no actor to configure.

2. Oxylabs

Oxylabs is a large data-collection company. Residential proxies, datacenter proxies, ready-made datasets, an unblocker, a hosted headless browser. Crawling has never been the main event, and in 2026 that's more true than it used to be.

The deterministic Web Crawler they used to document - max depth, URL regex filters, S3 delivery, scheduling - is gone from their developer docs. What replaced it is AI-Crawler, part of their AI Studio suite. You give it a starting URL and a natural-language prompt, and an agent decides which pages are relevant and extracts JSON or Markdown. There's also AI-Map, which enumerates the URLs on a domain - the closest thing anyone in this comparison offers to sitemap-style discovery.

It's a different tool with a different failure mode. A prompt-driven crawler is great when you want "all the pricing pages" and can't express that as a regex. It's worse when you want every page, and want the same crawl to return the same set next week.

Their separate Web Scraper API is still the mature product: $49/month for the Micro tier covering up to 98,000 results, $1.15 per 1,000 results for a generic site without JS rendering, $1.35 per 1,000 with it. Amazon and Google have their own rates. Free trial is 2,000 results. But that's a per-URL scraping product, not a crawler.

Pros

• Serious infrastructure behind it - proxy network, uptime, enterprise support, account managers
• AI-Crawler's prompt-based page selection genuinely handles vague requirements well
• AI-Map is useful on its own for URL discovery
• Python and JavaScript SDKs, plus an MCP server

Cons

• Whole-site crawling is now only available through the AI layer - no deterministic depth-and-regex crawler in the docs
• AI Studio doesn't publish a credits-per-page rate. Plans start at $12/month for 3,000 credits, but nothing in their docs maps a crawled page to a credit cost. I asked their own documentation assistant and it confirmed the figure isn't published. That makes budgeting a guess until you run a test.
• Crawling is one product among a dozen, and it shows in how much of the site is about proxies

3. Crawlbase

Crawlbase is a solid piece of infrastructure with a name that oversells one specific thing. It's on this list because people shortlist it when they're shopping for a crawler - and it's worth knowing, before you get to the invoice, that it isn't one.

Despite "crawl" in the name, it does not discover pages for you. The Crawling API takes one URL per request and returns that page. The Enterprise Crawler is asynchronous and high-volume, but you push the URLs in and get results back via callbacks. There are no depth or follow-links parameters in their crawling API docs. If you need to turn a domain into a page list, you build that part yourself.

What it's good at is the fetch layer. 140M residential IPs, 30 geographies, real browser rendering, and maintained scrapers for the sites that fight back hardest - Amazon, Walmart, Google SERPs, LinkedIn, eBay. Output now includes Markdown alongside HTML and structured JSON, which is new since I last looked. Custom headers and cookies are supported, so authenticated pages are on the table. SDKs cover seven languages.

Pros

• Best price at volume in this comparison, by a wide margin
• Maintained scrapers for high-value sites, updated by them when layouts change
• Cookies and custom headers, so logged-in pages are possible
• Markdown output alongside HTML and JSON
• Free tier of up to 5,000 requests with no card

Cons

• Not a crawler in the sense this post means. You bring the URLs.
• The pricing ladder is genuinely cheap in bulk but hard to predict: JS rendering and a per-domain complexity tier (standard, moderate, complex) apply multipliers that aren't published as numbers
• Structured extraction is scraper-template-first, which is a different mental model than crawl-and-clean

Pricing

Pay-as-you-go on a graduated ladder, charged per successful request: $3.00 per 1,000 for the first 1K, $2.00 for the next 10K, $0.60 for the next 100K, then $0.50, $0.10, $0.05, $0.04, and $0.02 at the very top. Their own worked example puts 100K requests at $76.40. Subscriptions start at $99/month for 200K credits.

4. Usescraper

Usescraper is gone.

I included it in earlier versions of this comparison as the indie option - simple, PAYG, Markdown output, and the one product here that supported sitemap-based crawling. As of September 2026, usescraper.com 301-redirects through layercode.com to toyo.ai, an unrelated AI assistant product. The docs.usescraper.com subdomain no longer resolves at all.

I'm leaving the section in rather than quietly deleting it, because if you're reading this post you may well have found Usescraper in an older comparison and be wondering where it went. It went away. Don't build on it.

Its useful feature - sitemap-first crawling - now has no clean equivalent in this list. Oxylabs' AI-Map is the nearest thing.

5. Apify

Apify is not an API you call to get a website back. It's a platform for building, hosting, and running crawlers, with a marketplace of roughly 68,000 pre-built ones ("actors") on top.

For this post's use case the relevant actor is Website Content Crawler - it crawls a site and extracts text specifically for feeding AI models and vector databases. It works well. But you're configuring an actor, not calling an endpoint, and the output shape is whatever that actor's author decided.

The tradeoff cuts both ways. If your requirements are unusual - click through a wizard, handle a login, parse a weird pagination scheme - Apify lets you write exactly that and run it on their infrastructure. No fixed API matches that. If your requirements are "give me this domain as Markdown," you're doing a lot of setup for something the other tools do in one request.

Cost is the other catch. Billing is per compute unit, so an inefficient crawl on a slow site costs more than an efficient one on a fast site, and you won't know which you have until you run it. Budget by experiment, not by spreadsheet.

If you landed here mainly because of Apify, we have a longer Apify alternatives comparison that goes deeper.

Pros

• Total control - if you can code it, you can run it
• Enormous actor library, including well-maintained ones for AI ingestion
• Handles logins, interactions, and multi-step flows that fixed APIs can't
• Generous-enough free tier to evaluate properly

Cons

• Steeper learning curve than everything else here
• Compute-unit billing makes costs hard to predict before you run something
• No plain "URL in, content out" endpoint - you're always choosing or writing an actor

Pricing

Free plan includes $5 of platform usage at $0.2 per compute unit. Starter is $19/month, Scale $199/month at $0.16 per CU, Business $999/month at $0.13 per CU. Concurrent runs, extra RAM, and proxies are priced separately as add-ons.

So which one

Four live options, one casualty, and no single winner:

1. WebCrawlerAPI - if you want the shortest path from a domain to clean Markdown for an AI knowledge base, with no subscription and no setup. Not if you need logins or you're crawling millions of pages a month.
2. Crawlbase - if you already have your URL list and want the cheapest reliable fetch layer at volume, or you need cookies and hard-site scrapers.
3. Apify - if your crawl is genuinely custom and you'd rather write code than fit into someone's parameter list.
4. Oxylabs - if you're already buying their proxies, or if prompt-driven page selection fits how you think about the problem better than regex does.
5. Usescraper - no longer an option.

I've tried to make this comparison as impartial as I can while also selling one of the products in it. If something above is wrong, outdated, or unfair, tell me: hello@webcrawlerapi.com. You can try WebCrawlerAPI at webcrawlerapi.com without a card.