WebCrawlerAPI
Guides16API reference19SDKs4Integrations6

Guides/Markdown & scrape/Markdown

LLM-cleaned Markdown

Get the clean, readable text of a page — with the clutter removed by an LLM.

Open in Claude Open in ChatGPT

A normal scrape gives you the whole page as markdown: the article or product info you actually want, plus everything else on the page — navigation menus, sidebars, footers, cookie banners, newsletter sign-up forms, ads, and related-content widgets.

LLM-cleaned markdown skips all of that. An LLM reads the page and returns just the main content, ready to read, store, or feed into another model — no post-processing needed on your end.

What gets cut

  • Navigation menus, sidebars, and footers
  • Cookie consent banners and privacy notices
  • Newsletter and email sign-up forms
  • Ads and promotional widgets
  • "Related articles" / "You might also like" sections
  • Comment sections and social share buttons

What's kept

The main content — headings, body text, lists, and images that belong to the article or page itself — in clean markdown.

If the LLM cleaning fails for any reason, you still get a result: the endpoint falls back to plain, uncleaned markdown instead of erroring out.

Best for

Pages where the content is dense, readable text — news articles, blog posts, and documentation pages — since that's where an LLM does the most useful work separating content from clutter. For pages that are already mostly just the content (or where you don't need cleaning), plain /scrape markdown is cheaper.

Pricing

Billed per page, depending on your plan. See the pricing page for the current rates.

Quick example

bash
curl --request GET \
  --url 'https://api.webcrawlerapi.com/markdown?url=https://example.com/blog/some-article' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Response:

json
{
  "success": true,
  "markdown": "# Some Article\n\nThis is the cleaned main content of the page..."
}

API reference