What is the best data format for scraped data?

Answer

The best format depends on how you plan to use the data. CSV is simple and works well for tabular data and quick analysis. JSON handles nested structures and is easy to pass between services. Parquet is efficient for large datasets and analytics. Choose a format that fits your storage, query, and downstream tooling.

If the downstream consumer is an LLM — a knowledge-base search pipeline, an agent, a summarization job — the calculus changes. Markdown is usually the better default, and the reason isn't style preference, it's tokens and structure.

Why markdown beats raw HTML for LLM ingestion

Raw HTML is expensive to feed into a model. A typical page is 60-80% markup: <div> wrappers, class names, inline styles, <script>/<style> blocks, tracking attributes. None of that carries meaning for the model, but every tag still costs tokens. Converting to markdown routinely cuts token count by half or more for the same visible content — which matters for cost and for how much context window is left for actual reasoning.

Markdown also keeps what plain text loses: structure. Rendered as ## Section, ### Subsection, and lists, it preserves the heading hierarchy the author already used to organize the content. That hierarchy is what a chunking step relies on — split on headings and you get chunks that map to actual topics, not arbitrary character-count slices that cut a sentence or a table in half. Plain text throws this away; HTML buries it under noise.

What to strip, and why it poisons chunks

Boilerplate — nav bars, footers, cookie banners, newsletter widgets, related-article rails — repeats on every page with nearly identical wording. If it survives into your markdown, it doesn't just waste tokens, it degrades retrieval: an embedding for a chunk that's 40% "Accept cookies / Sign up for our newsletter" drifts toward that boilerplate's meaning instead of the actual content, so it scores worse for the queries it should match. A handful of pages with unstripped boilerplate can drag down answer quality across a whole search index, since the noise repeats page after page.

A clean extraction isolates main content and drops the rest:

<!-- Strip these before indexing -->
- Navigation menus and breadcrumbs
- Footer links and legal boilerplate
- Cookie/consent banners
- Ads and promotional widgets
- "Related posts" rails

What's left should read like an article: headings, paragraphs, lists, and tables, in order, with no layout chrome.

When JSON or JSONL is the better answer

Markdown is the right call for reading and reasoning over prose. It's the wrong call when extracting structured fields — price, SKU, rating, address, publish date — into a schema you'll query or join elsewhere. For that, JSON (or JSONL for streaming records into an embeddings pipeline) is more reliable: typed fields, no re-parsing prose for a number, and it plugs into a vector store's metadata alongside the embedding instead of living inside the chunk text.

A practical split: markdown for body content you'll chunk and embed, JSON for structured facts you'll filter or aggregate on. Many pipelines use both — markdown for article text, a JSON sidecar for title, author, date, and canonical URL as metadata on each chunk.

When CSV or Parquet still win

Neither markdown nor JSON fits bulk tabular data. Scraping thousands of product listings or directory entries for analytics is still a CSV or Parquet job — Parquet especially at scale, since columnar storage and compression make analytical queries far cheaper than scanning row-oriented JSON. The consumer here is a data warehouse, not a model.

How format choice affects retrieval quality

The format you pick shapes the whole pipeline downstream. Token-efficient markdown means more of a page fits in one chunk without truncation. Preserved headings mean chunk boundaries land on topic boundaries, not mid-sentence. Stripped boilerplate means embeddings represent the actual content, not an average with site chrome. Get this wrong and no amount of tuning downstream fully recovers it — bad chunks give the model bad context, and a model reasoning over bad context gives a bad answer regardless of ranking quality.

If you're feeding scraped pages into an LLM pipeline and don't want to build the HTML-to-markdown cleanup yourself, WebCrawlerAPI returns clean markdown for any URL — boilerplate stripped, headings intact — so you can go straight to chunking and embedding instead of writing a parser first.