A normal scrape gives you the whole page as markdown: the article or product info you actually want, plus everything else on the page — navigation menus, sidebars, footers, cookie banners, newsletter sign-up forms, ads, and related-content widgets.
LLM-cleaned markdown skips all of that. An LLM reads the page and returns just the main content, ready to read, store, or feed into another model — no post-processing needed on your end.
What gets cut
- Navigation menus, sidebars, and footers
- Cookie consent banners and privacy notices
- Newsletter and email sign-up forms
- Ads and promotional widgets
- "Related articles" / "You might also like" sections
- Comment sections and social share buttons
What's kept
The main content — headings, body text, lists, and images that belong to the article or page itself — in clean markdown.
If the LLM cleaning fails for any reason, you still get a result: the endpoint falls back to plain, uncleaned markdown instead of erroring out.
Best for
Pages where the content is dense, readable text — news articles, blog posts, and documentation pages — since
that's where an LLM does the most useful work separating content from clutter. For pages that are already mostly
just the content (or where you don't need cleaning), plain /scrape markdown is cheaper.
Pricing
Because it involves an LLM call, LLM-cleaned markdown costs more per page than a regular scrape. The extra charge is a flat, fixed amount per page — the same at every plan tier. See the pricing page for the current rate.
Quick example
curl --request GET \
--url 'https://api.webcrawlerapi.com/markdown?url=https://example.com/blog/some-article' \
--header 'Authorization: Bearer YOUR_API_KEY'Response:
{
"success": true,
"markdown": "# Some Article\n\nThis is the cleaned main content of the page..."
}API reference
- GET/POST /markdown — full request and response reference