How to Extract Article Content From Any Web Page (After Trying Everything Else)

I spent way more time on this than I want to admit. Every time I thought I'd cracked "extract the article, drop the junk," some site would come along and prove me wrong. Here's the actual order I tried things in, what broke, and where I landed.

Why I even needed this

I was building an AI knowledge base — feed it URLs, get back semantic search over it. Simple in theory: crawl a page, turn it into text, index it.

Except the page isn't the article. It's the article plus a sign-in form, a footer, three levels of nav, a cookie banner, and an ad slot that somehow always loads before the actual paragraph you wanted. Convert that whole mess to text or Markdown and all of it comes along for the ride — I'd already gone down the HTML vs. cleaned text vs. Markdown rabbit hole picking an output format for the knowledge base, and none of those formats solve this problem on their own — they just change what the junk looks like once it's in there.

Feed that into an LLM's context and you're not helping it — you're paying to confuse it. So before any of the "AI knowledge base" part could work, I had to solve a much dumber-sounding problem first: how do you keep just the article?

First attempt: just write selectors

This is where everyone starts, me included. Open the page, inspect it, find the <div> holding the article, grab that.

It works. For that one page. Then I pointed it at the next site and watched it grab the "related posts" sidebar instead, because on that site the sidebar and the article body shared the exact same class name. A WordPress post wraps its content in <article class="post-1234"><div class="entry-content">. Blogspot wraps it in <div class="post hentry"><div class="post-body">. Some custom CMS I hit wrapped it in <div id="chapterContent"> with no other useful hook anywhere on the page. Write a selector list long enough to cover all three and the next site you crawl invents a fourth pattern, and you're back to square one — except now you also have a pile of site-specific rules to maintain forever.

I gave up on selectors fast. Not because it's hard to write one — it's that there's no selector that generalizes, and generalizing was the entire point. This is the same wall I hit cleaning crawled data with BeautifulSoup: fine for one known page structure, a maintenance burden the moment you point it at a site you don't control.

Second attempt: just keep the visible text

Next idea: forget structure, just keep whatever text is actually visible on the page and throw away hidden elements. Sounded reasonable for about a day. Headers are visible. Footers are visible. That "sign up for our newsletter" box is extremely visible, usually in a contrasting color specifically so you can't miss it. This approach didn't fail loudly, it just quietly didn't solve the problem — I still had junk, it was just visible junk now instead of invisible junk.

Third attempt: heuristics

Then I tried scoring things — word density, link-to-text ratio, that kind of rule. The idea being: real article paragraphs are dense with text and light on links, nav and ad blocks are the opposite. I hand-rolled a few of these rules myself before realizing I was slowly reinventing something that already existed and was better tuned than anything I'd write in an afternoon: Readability.js, the algorithm Firefox's reader mode is built on.

Fourth attempt: Readability.js — closest I got before the LLM

This one actually worked, for a while. It's genuinely good at blog posts and news articles, which makes sense — that's exactly the shape of content it was built to score.

I shipped it. Called it "Main Content Only," a toggle you could flip and get clean article text back. Then people started pointing it at landing pages, and it fell apart in a way that was worse than not having the feature at all: it would confidently strip out real content because that content happened to live in a tag Readability.js doesn't associate with articles — not a <p>, not whatever it expects. Landing pages just don't have the word-density profile of a news article, and the algorithm has no idea what to do with that. It doesn't fail safe, it fails silently and returns something that looks plausible but is missing half the page.

I'd set the expectation that "Main Content Only" meant "works on any page," and it didn't. So I turned it off. That was a genuinely annoying decision to make — the feature worked, just not universally, and "works most of the time" isn't a thing you can ship a toggle for. If you want the deeper mechanics of how it scores content, and a Rust-based alternative I tested afterward for the same job, I wrote both of those up separately: the Readability.js breakdown and dom-smoothie, a Rust alternative.

Here's the whole run, side by side:

What actually worked: handing it to an LLM

By this point I'd tried every rule-based trick I could think of, and every one of them had a category of page it quietly failed on. The thing that finally stopped breaking was giving up on rules entirely and handing the page to an LLM.

LLMs are built for text, not DOM structure. They don't care what tag something sits in — they can tell a lead paragraph from a sign-up prompt the same way a person skimming the page would.

The prompt does most of the work. Mine looks like this:

Return the complete text of this news article as plain text: the lead paragraph and all following body paragraphs, in order, with Markdown and links stripped. Exclude ads, navigation, subscription or sign-up prompts, image captions, and related links. Do not summarize or rewrite.

Content: <PAGE MARKDOWN CONTENT>

Two things about running this in production that nobody warns you about:

• Cost and context length add up fast. Every page is a request now instead of a free regex match. Long pages — docs, long-form articles — can blow past a smaller model's context window, so you end up chunking the page and stitching the output back together. Chunk in the wrong spot and you cut a paragraph clean in half.
• "Do not summarize" is a request, not a guarantee. I've watched a model quietly paraphrase a paragraph instead of returning it word for word, even with that instruction sitting right there in the prompt. If you actually need the exact source text, don't just trust the instruction — add a cheap verification step, like checking the output length or a substring match against the source, and re-run it if it's off.

If you want to do this yourself, without an API

You don't need our API to try this. The shape is the same wherever you run it: fetch the page, turn it into Markdown or clean text, send it to any LLM with the prompt above.

const res = await fetch(pageUrl);
const html = await res.text();
const markdown = htmlToMarkdown(html); // any HTML-to-Markdown lib

const completion = await llm.chat({
  messages: [{
    role: "user",
    content: `Return the complete text of this news article as plain text...\n\nContent: ${markdown}`,
  }],
});

That's genuinely fine for a handful of pages. It starts hurting at the exact same points everything above did: sites behind CAPTCHAs or rate limits, pages that need JS to render before there's anything to extract, plus the chunking and verification problems from the last section. At some point you're not writing an extraction script anymore, you're maintaining a small crawling pipeline just to keep it fed.

Where I ended up: an API for it

That's the point where it stopped being worth doing myself every time, which is basically why Webcrawler Agent exists — it's become one of the main things people use it for. It extracts the article and drops the junk automatically, CAPTCHAs and rate limits included, so I don't have to think about chunking or retries anymore.

curl -X POST https://api.webcrawlerapi.com/v1/agent \
  -H "Authorization: Bearer <API_TOKEN>" \
  -d '{
    "prompt": "Extract main blog post article content",
    "urls": ["https://webcrawlerapi.com/blog/how-to-find-a-company-website"]
  }'

It's not limited to articles, either. Point it at any page and ask for one specific thing — contact details, pricing, whatever — and pipe that straight into whatever you're building.

FAQ

Can I use CSS selectors to extract article content? For one site you control, sure — inspect it, find the container, write the selector. It stops working the moment you need this across sites you don't control, because the same class or tag holds real content on one site and junk on another. I tried this first and dropped it fastest.

Does Readability.js work on landing pages? Not reliably, and it took shipping a feature and turning it back off to learn that. It's tuned for the word-density profile of blog posts and articles. Landing pages don't share that shape, and Readability.js can silently cut real content sitting outside the tags it expects. See the full breakdown if you're extracting with it in JS.

What's the cheapest way to extract article text at scale? Readability.js or something like it, if every page you're hitting is genuinely blog-post-shaped and you can live with occasional misses. Once you're mixing page types, or the misses actually cost you something, an LLM-based extraction is more reliable — you're trading a free local computation for a per-page request.