Markdown Chunking in Python: Header Splitting Measured Against Fixed and Recursive Windows

Markdown chunking is text chunking with extra information: headings say where a topic starts. This post compares five chunkers on 130 crawled pages from the FastAPI and htmx docs, scored on 93 questions. The short answer: no chunker beat the others on recall by more than the noise, and prepending the heading path to the embedded text was worse on htmx and never better. Use LangChain's RecursiveCharacterTextSplitter with the markdown separators and is_separator_regex=True (code in Step 2, B). It is the default because it was nominally best on recall@1 and MRR, never worse than fixed windows, and is six lines. Its lead is not statistically clear. If answers must cite a section, use header sections with a 400-token cap and keep the heading path as metadata, not in the embedded text.

This is the measured version of the chunking step in the website to markdown knowledge base guide. The overview of chunking strategies covers the options in general.

Results at a glance

Tests run and numbers read November 1, 2026. Embeddings: openai/text-embedding-3-small through OpenRouter. 130 pages (80 FastAPI, 50 htmx), 93 questions (46 and 47). The letters A to E are defined in Step 2.

Pooled over both sites. B led at recall@1, but its gap to the others is inside the noise (paired bootstrap, below). Two results are clear. D is worse on htmx, and the capped header chunkers return about 40% fewer tokens in the top 5 than fixed windows (1,103 against 1,826).

Markdown chunking on crawled docs pages: what breaks

A crawled docs page arrives as markdown with the title as #, topics as ## and ###, and code in fenced blocks. A header chunker treats each heading as a boundary and keeps the heading text as a label. That works when sections run a few hundred tokens. Real docs break it in three ways:

• Headings inside code fences. A # comment line in a Python block looks like a heading to a naive regex.
• Huge sections. One FastAPI section ran to 7,752 tokens, with no heading anywhere inside it.
• Tiny sections. A heading followed by one line becomes a chunk of 10 or 20 tokens that can match on its own heading words.

The diagram cuts one FastAPI page (/tutorial/body-updates/, 2,082 tokens) four ways. Red dots mark chunks that start or end inside a code block.

Step 1: Get markdown with the source URL

The pages come from POST /v1/crawl (crawl API docs), polled through the job endpoint until status is done. Each finished item has original_url, title and a markdown_content_url. Full crawl-and-save code is in crawl a website to markdown files and the Python knowledge base build. The core (excerpt of crawl.py) is:

body = {
    "url": "https://fastapi.tiangolo.com/tutorial/",
    "whitelist_regexp": r"^https://fastapi\.tiangolo\.com/(tutorial|advanced)/",
    "items_limit": 80,
    "output_formats": ["markdown"],
    "respect_robots_txt": True,
    "max_age": 0,  # skip the 7-day cache, fetch fresh
}
r = http.post("https://api.webcrawlerapi.com/v1/crawl", json=body)  # Authorization: Bearer YOUR_API_KEY
job_id = r.json()["id"]
# poll GET /v1/job/{job_id}, then for each item with status "done":
#   write {"url": item["original_url"], "title": item["title"], "markdown": <text from markdown_content_url>}

The htmx crawl started at /docs/ with a whitelist for docs, reference, attributes and examples (URL filters). Both ran with respect_robots_txt: true. max_age: 0 skips the 7-day cache (caching docs). main_content_only stayed at its default of false: earlier tests found identical token counts with and without it on 10 htmx pages (website to markdown guide) and identical markdown on 19 of 20 pages (token count post), which does not make it neutral everywhere. The cleaning guide documents clean_selectors for removing page elements, which this run did not use.

Step 2: Five chunkers

All five share one setting. Sizes are counted in tokens with tiktoken and cl100k_base, the encoding behind text-embedding-3-small (see token counts of web pages for why tokens, not characters). The cap is 400 tokens (before any heading-path prefix) and the overlap 60 (15%), with langchain-text-splitters 1.1.3 and tiktoken 0.14.0. The full file is chunkers.py.

A. Fixed-size token windows with overlap

The baseline ignores structure. Encode the page, slide a 400-token window forward 340 tokens at a time, decode each window. Overlap keeps a sentence cut at a boundary whole in one of the windows. The snippet is a simplified form of fixed() in chunkers.py, which also tracks character offsets.

toks = ENC.encode(md, disallowed_special=())
step = CAP - OVERLAP              # 340
for i in range(0, len(toks), step):
    window = ENC.decode(toks[i:i + CAP])

On the sample page, 4 of 6 windows start or end inside a code block.

B. LangChain RecursiveCharacterTextSplitter

The recursive splitter tries a list of separators in order and falls back to the next one when a piece is still too long. The markdown separators put heading marks first, then fences, then blank lines. They are regexes, so the splitter needs is_separator_regex=True.

from langchain_text_splitters import Language, RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base", chunk_size=400, chunk_overlap=60,
    separators=RecursiveCharacterTextSplitter.get_separators_for_language(Language.MARKDOWN),
    is_separator_regex=True,  # without this the heading separators are escaped and never match
)
pieces = splitter.split_text(md)

from_tiktoken_encoder was used because from_language alone counts characters, which would put the cap in a different unit from the other chunkers. On the sample page, 2 of 8 chunks start or end inside a code block.

C. MarkdownHeaderTextSplitter: pure header split

MarkdownHeaderTextSplitter is the class most people mean by "split markdown by headers". It does not inherit from TextSplitter, it has no size parameter, and it returns Document objects with the heading path in metadata.

from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")],
    strip_headers=False,  # keep the heading line inside the chunk text
)
docs = splitter.split_text(md)
# docs[1].metadata -> {"h1": "Body - Updates", "h2": "Update replacing with `PUT`"}
# (after stripping FastAPI's [¶](...) permalink from the raw heading text)

strip_headers=False keeps the heading in the text. Without a cap, a section is as big as the author made it: median 194 tokens, 95th percentile 976, largest 7,752 (close to the 8,191-token input limit of text-embedding-3-small).

D. Header split, size cap and heading path

This chunker is plain Python and the only one of my own design, which is why its parameters were frozen before scoring. The steps:

1. Split at # to ### headings, skipping lines inside fenced code.
2. Merge a section under 80 tokens into the previous one, if the merged size stays within 400. A tiny first section (a "Skip to content" link) merges forward.
3. Send any section over 400 tokens to the recursive splitter with 60 tokens of overlap.
4. Prepend the heading path (Body - Updates > Partial updates with PATCH) to the text that is embedded. The stored body stays clean.

Across all 130 pages, 48% of D chunks start or end inside a code block, all from forced splits of long sections (A: 63%, B: 53%). C never does (0 of 973), since it never cuts inside a section. Not handled: a code block or table over 400 tokens is still cut, and level 4 headings are not boundaries.

E. D without the heading path

E has the same boundaries as D and embeds the body only, which isolates the prefix from the boundaries. This is the function to copy if answers need a section citation. It is chunk_page.py, checked against chunkers.py on all 130 pages with identical chunks.

import re
import tiktoken
from langchain_text_splitters import Language, RecursiveCharacterTextSplitter

ENC = tiktoken.get_encoding("cl100k_base")
CAP, OVERLAP, MERGE_BELOW = 400, 60, 80
FENCE = re.compile(r"^\s*(```|~~~)")
HEAD = re.compile(r"^(#{1,3})\s+(.+?)\s*#*\s*$")
LINK = re.compile(r"\[([^\]]*)\]\([^)]*\)")
SPLITTER = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base", chunk_size=CAP, chunk_overlap=OVERLAP, is_separator_regex=True,
    separators=RecursiveCharacterTextSplitter.get_separators_for_language(Language.MARKDOWN))


def tokens(text):
    return len(ENC.encode(text, disallowed_special=()))


def clean_heading(h):
    """Drop markdown links from a heading (FastAPI adds a [¶](...) permalink to every heading)."""
    return LINK.sub(r"\1", h).replace("¶", "").strip()


def chunk_page(md, url, title=""):
    # 1. sections at # to ### headings, ignoring lines inside code fences; keep the heading path
    secs, stack, in_fence, pos, start, path = [], {}, False, 0, 0, ""
    for line in md.splitlines(keepends=True):
        in_fence ^= bool(FENCE.match(line))
        m = None if in_fence else HEAD.match(line.rstrip("\n"))
        if m:
            if pos > start:
                secs.append([path, start, pos])
            lvl = len(m.group(1))
            stack = {k: v for k, v in stack.items() if k < lvl}
            stack[lvl] = clean_heading(m.group(2))
            path, start = " > ".join(stack[k] for k in sorted(stack)), pos
        pos += len(line)
    if pos > start:
        secs.append([path, start, pos])
    # 2. merge sections under MERGE_BELOW tokens: into the previous one, or forward if the previous one is tiny
    merged = []
    for s in secs:
        small = tokens(md[s[1]:s[2]]) < MERGE_BELOW
        prev_small = bool(merged) and tokens(md[merged[-1][1]:merged[-1][2]]) < MERGE_BELOW
        if merged and small and tokens(md[merged[-1][1]:s[2]]) <= CAP:
            merged[-1][2] = s[2]
        elif prev_small:
            merged[-1] = [s[0], merged[-1][1], s[2]]
        else:
            merged.append(s)
    # 3. cap: sections over CAP tokens go through the recursive splitter
    out = []
    for path, s, e in merged:
        body = md[s:e]
        for piece in ([body] if tokens(body) <= CAP else SPLITTER.split_text(body)):
            if piece.strip():
                out.append({"url": url, "section_path": path or title, "body": piece, "tokens": tokens(piece)})
    return out

The block is the whole file, so it runs as pasted. It needs tiktoken and langchain-text-splitters 1.1.3.

The chunk record: overlap and source URLs

Every chunker in chunkers.py returns the same record:

{"url": ..., "title": ..., "section_path": ..., "char_start": ..., "char_end": ...,
 "body": ..., "text": ..., "tokens": ...}

body is the page text. text is what gets embedded, and it differs from body only for D (path added). char_start and char_end locate the chunk in the page, which the scoring needs. The url and section_path pair is what an answer cites, as in the answer step of the Python knowledge base build. Overlap is applied everywhere for A and B, and only inside forced splits for D and E. No overlap ablation was run.

How the test was run

Corpora. FastAPI docs (/tutorial/ and /advanced/, MIT licensed; median page 2,172 tokens) and htmx docs (docs, reference, attributes, examples; median 399 tokens, longest 21,335). FastAPI's robots.txt had no directives at the time of the crawl, and htmx allows everything. The htmx pages come from the same site as the guide's earlier 20-question test, which had fixed windows ahead of header chunks on rank-1 page hits (15 questions against 12), so htmx is not independent confirmation. FastAPI is the independent check.

Questions. A language model (openai/gpt-4.1-mini via OpenRouter, temperature 0.3) read one whole page at a time and wrote four questions each, plus a one-line answer quote copied from the page. It never saw a chunk or a section split. 16 pages per site were sampled (seed 8, at least 300 tokens). Of 128 questions generated:

• 21 were dropped because the quote was not found verbatim on the page.
• 8 were dropped because the quote also appeared on another page.
• 5 were dropped because the question repeated a heading.
• 94 were kept.

I read 24 of the 94 (log in hand_check.json). One referred to "the example" and was rewritten, and one was trivia about a code comment and was dropped, which left 93. The questions echo page wording, which favors lexical overlap. They are not support tickets, so recall on real tickets may be lower for every chunker.

Fixed before scoring, with one correction. The chunker parameters were saved with a hash at 11:57 UTC and the question set at 11:59 UTC on November 1, before any score existed. After the first scoring run, a review found that the recursive splitter's markdown separators were being escaped, so the heading separators never matched. The flag is_separator_regex=True was added, nothing else changed (same eval set, same parameters), and B, D and E were rescored. A and C do not use that splitter. The first-run results are kept as results-run1.json. The fix moved B from 54.8% to 63.4% recall@1 and D from 58.1% to 55.9%. Every number here is from the second run.

Scoring. The 93 questions and every chunk were embedded with openai/text-embedding-3-small and ranked by cosine similarity in numpy, top 5, with no database (for storing more than that, see vector databases for a knowledge base). Two scores per question:

• Page hit: a chunk from the gold page.
• Section hit: a chunk that covers at least half of the answer quote's character span (a chunk holding the whole quote always counts).

The section rule is the useful one. Page recall at rank 5 was 90.3% to 94.6% for every chunker and hides most differences. Also recorded: MRR@5, chunk sizes, and mean tokens in the top 5, which matters because header-only chunks have no cap.

Statistics. Questions come in groups of up to four from the same page, so a paired bootstrap resampled pages (1,000 resamples) within each site and compared chunkers on the same resamples. At 46 and 47 questions per site, one question is about 2 points. The pooled intervals were 11 to 24 points wide, and 12 to 45 per site. Gaps under about 10 points are treated as noise throughout.

Spend. About $0.33: $0.26 for the crawls, $0.04 for question generation and $0.03 for embeddings (1,684,600 tokens at list price, including the first run's B, D and E chunks that the fix replaced).

Results: no winner on recall, and the heading path never helped

Section recall by site (percent of questions):

On FastAPI, B, D and E lead at rank 1 over fixed windows. On htmx they do not, and D falls clearly behind.

Differences in section recall (percentage points, 95% interval from the paired bootstrap; * means the interval excludes zero):

The rest are in results.md. Of 54 intervals, 8 excluded zero. All 8 involve D (7 on htmx, 1 pooled: D minus B on recall@5), and all point against D, so the penalty for the prefixed chunker looks real on htmx. By chance alone, two or three scattered intervals would be expected. E minus B was added after the fact: B was ahead on htmx MRR (E minus B -0.048, interval -0.081 to -0.019), though not on pooled recall@1 (-4.3, interval -11.4 to +1.9).

Compared with the earlier 20-question test. On htmx, page recall@1 here was 63.8% for fixed windows and 59.6% for header-only chunks (30 and 28 of 47 questions). The direction matches the earlier test, but the gap is two questions, so neither test shows a real difference.

Chunk sizes and cost

Embedding costs about five cents per 1,000 pages for every chunker. Context size differs: fixed windows hand the model 1,826 tokens for five hits, E 1,103 and B 1,291. Header-only (1,458) has a wide spread: in 15 of 93 questions the top 5 contained a header chunk over 800 tokens.

Where each chunker failed

• Neighboring section, same page. For "What does FastAPI's OAuth2PasswordBearer dependency look for in the request?", fixed windows hit at rank 1 with a 307-token window. D ranked a 181-token chunk from the same page first ("Security - First Steps > OAuth2PasswordBearer > Use it"), but the answer was in the "What it does" section next to it. The heading words matched the question, the answer text did not.
• A small chunk that wins on its label. For "How do you import Field when using Pydantic models with FastAPI?", header-only ranked a 111-token chunk from the "Form Models" page first and found the right section at rank 3. D put a 48-token chunk first, a section that is mostly its own heading path. That one is a hit, but a 48-token chunk is thin context for an answer.
• Questions with two good pages. For "How can I convert input data to JSON-compatible format before saving it with FastAPI?", every chunker ranked the JSON Compatible Encoder page first. The gold quote was on the body-updates page, but the encoder page is a fair answer too, so recall here is a lower bound for all five.
• Nobody found it. 7 of 93 questions had no section hit in the top 5 for any chunker. 31 were hit at rank 1 by all five. Header-only returned a chunk under 80 tokens first in 12 questions and hit in 5 of them.

The heading path (D against E)

D and E share boundaries, so the difference is the prefix. Pooled recall@1 was 55.9% with it and 59.1% without. Paired by question, D alone hit in 4 cases and E alone in 7. On htmx the gap was large (recall@5 74.5% against 85.1%) and the interval excluded zero. One possible reason is that heading words count twice (heading line and prefix). That was not tested.

What to use

On two docs sites and 93 questions, recall did not separate the chunkers except the one with the heading path, so choose by the shape of the output and what the answer has to cite.

• Default: B, the recursive splitter with markdown separators and is_separator_regex=True. It was nominally best on recall@1 and MRR, uses about 29% fewer tokens in the top 5 than fixed windows, and is six lines of LangChain.
• Exception: when an answer must cite a section, use E (chunk_page). Recall was within noise of A, though behind B on htmx MRR (-0.048), the top 5 carries about 40% fewer tokens than fixed windows, and section_path comes with every chunk. Expect lower htmx recall@1 than B (48.9% against 55.3%). Keep the path out of the embedded text.

Plain text exports and PDFs converted without structure give a header splitter nothing to split on, and B handles them with no changes.

Not tested: semantic chunking and LLM-based chunking (see chunking strategies for those), rerankers, hybrid search, chunk sizes other than 400, overlaps other than 60, and embedding models other than text-embedding-3-small. A comparison of embedding models is planned for this series.

Gotchas found on crawled markdown

• Markdown separators are regexes. get_separators_for_language(Language.MARKDOWN) returns patterns such as \n#{1,6} . Without is_separator_regex=True they are escaped, the heading separators never match, and the splitter behaves like a fence, paragraph and line splitter. The first run of this test had that bug.
• Permalinks in headings. FastAPI headings come through as # First Steps[¶](https://fastapi.tiangolo.com/tutorial/first-steps/#first-steps "Permanent link") on 80 of 80 pages. Left alone, every heading path carries a URL. clean_heading() strips markdown links from labels before they go into a path.
• A "Skip to content" line before the title. All 80 FastAPI pages start with one, and header-only made it a chunk of its own on 77 of them. D and E merge it forward. C keeps it.
• LangChain's start_index is unreliable when text repeats. 127 of 1,389 recursive chunks came back out of order or negative with add_start_index=True. The chunkers here search forward from the previous start instead.

Conclusion and next steps

Header splitting did not beat a well-configured recursive splitter on recall, and the heading path in the embedded text hurt on htmx. What headers give is a section to cite, at no measurable cost in recall once the path stays in metadata. The full pipeline around this step is in the website to markdown knowledge base guide.

• Rerun the test on your own pages and questions: scripts, question set, results and saved console output are in the code folder. Pages are not included. Recrawl them with crawl.py.
• Embed and answer with the chunks: build a knowledge base from a website in Python.
• Crawl your own docs site to markdown with the crawl API (parameters in the crawl docs), then run evaluate.py against it with your own questions.