RAG Chunking Strategy, Decided by a 50-Pair Eval

Every chunking guide converges on the same numbers eventually: about 512 tokens, 10 to 20 percent overlap. Almost none of them show the measurement behind those numbers, because there isn't one. A RAG chunking strategy is an empirical property of your corpus and your query mix, and the honest answer to how to choose a chunking strategy for RAG is to measure it. Build a small labeled retrieval eval, compute three signals from your own data, and let recall numbers plus token arithmetic pick the branch. Sometimes the measurement tells you not to re-chunk at all.

Where the Standard Chunking Advice Goes Wrong

The dominant format is the decision tree: a flowchart asks about your documents and walks you to a strategy. The better examples, like Pinecone's chunking strategies overview, describe the taxonomy accurately. The genre's weakness is that the branches are rarely tested against any data. Greg Kamradt's five levels of text splitting notebook popularized the ladder from character splitting to semantic splitting, and hundreds of guides copied the ladder without the one thing that would make it useful, a way to tell which rung your corpus actually needs.

Four variables decide, and most decision trees mention at most one:

• Corpus structure. Markdown, HTML, and code carry explicit topic boundaries. PDF-extracted prose and call transcripts carry none.
• Query granularity. Users asking pinpoint questions ("what is the default timeout") need different chunks than users asking multi-hop ones ("did the fix for X also close the issue from March").
• Embedder limits. Your embedding model's context window bounds every strategy that wants longer units.
• Unit cost. Index size, preprocessing passes, and retrieval tokens differ by 2x or more across strategies.

No single default can be right on all four axes at once. When a teammate asks why you chose 512, a blog post is not a defensible answer in review.

Build the Eval Before You Pick a RAG Chunking Strategy

Evaluating chunking on your own corpus takes an afternoon. Fifty labeled question-to-source pairs are usually enough, because you are ranking a handful of variants on one fixed corpus, not estimating a population statistic to three decimals. Strategy-to-strategy recall gaps on the same corpus tend to dwarf the sampling noise at that scale.

1. Sample 50 real questions from query logs, support tickets, or the questions users actually ask. Avoid writing them yourself; invented questions carry invented phrasing.
2. Label the gold source for each question, meaning the document or span that contains the answer. Two people labeling 50 items takes under an hour.
3. Freeze everything except chunking. Same retriever, same embedding model, same top-k, reranking off. Change one chunking variable at a time.
4. Score recall at k against gold sources. The core question of a recall at k retrieval evaluation is simply whether the correct document appeared in the top k. Precision at k and token cost per query are your secondary gates; RAGAS context recall is the reference definition if you want an off-the-shelf implementation.

The scorer itself is a few lines:

def recall_at_k(hits, gold, k):
    # hits: question -> ranked list of source doc ids
    # gold: question -> set of acceptable doc ids
    return sum(bool(gold[q] & set(h[:k])) for q, h in hits.items()) / len(hits)

Your results table should end up shaped like this. The cost columns are deterministic arithmetic for a 1M-token corpus; the metric cells fill in from your run.

Strategy	recall@10	precision@10	Vectors	Retrieval tokens per query
Recursive 512, 10% overlap	yours	yours	~2,169	~4,100
Structure-aware, 600 cap	yours	yours	~1,800	~4,800
Parent-child, 128/512	yours	yours	~7,813 child plus stored parents	~2,000

Three Corpus Signals That Decide the Branch

Each signal takes minutes to compute and points at exactly one branch.

Structure density. What share of documents carry explicit structure? Compute the fraction that is markdown, HTML, or code, and the average heading count per 1,000 tokens. High density means heading boundaries already encode topic shifts, and a splitter that ignores them is discarding free labels. Low density is what pushes you toward semantic chunking or contextual chunk headers.

Query granularity versus chunk granularity. Label 30 real queries as pinpoint (one fact, one place) or multi-hop (synthesis across sections or documents). This mismatch is the leading chunking failure mode: pinpoint questions retrieved from big chunks drown the needle in padding, while multi-hop questions retrieved from tiny chunks never assemble the full picture.

Embedder context limit. Check the window and pricing in the OpenAI embeddings guide or your provider's equivalent. A short window forces splitting before semantics. A long window, 8k and up, keeps a fifth path open where you skip splitter decisions entirely.

Decision Gates for Six Chunking Strategies

This table keys the six common chunking strategies for RAG to the signals above, and each row carries the failure signature you will see in the eval when it is the wrong branch.

Strategy	Ship it when	Failure signature
Fixed-size	Uniform prose, no structure, quick baseline	Cuts tables, code, and sentences mid-thought; answers straddle boundaries and lose their subject
Recursive text splitting	Mixed prose, weak headings, your default baseline	Respects paragraph edges but is blind to topic shifts; one size cannot serve two query granularities
Structure-aware chunking	High structure density, real headings and sections	Fake headings from PDF extraction; oversized sections still need a size cap
Semantic chunking	Low structure, drifting prose, boundaries worth paying for	Roughly doubles preprocessing; thresholds merge unrelated text or fragment related text
Parent-child chunking	Granularity mismatch: pinpoint queries, multi-hop answers	Roughly doubles stored tokens; oversized parents dilute, undersized children embed context-free
Late chunking	Long documents that fit a long-context embedder	Gated on the embedder window and pooling access; full-document embedding cost

Recursive text splitting remains the right baseline for most corpora because it respects paragraph and sentence edges while staying cheap; the RecursiveCharacterTextSplitter documentation is the canonical reference for its separator hierarchy. For structure-aware work, Unstructured chunk-by-title partitioning splits at titles that survive document extraction, which is the version you want when the source is messy PDF or Office output.

Worked example, markdown docs

An internal engineering wiki that is 90 percent markdown with several headings per 1,000 tokens, queried mostly for pinpoint facts. The structure signal is strong, so split at H2 and H3 with a 600-token cap. Semantic chunking here spends roughly double the preprocessing to approximate boundaries the headings already state. Expect structure-aware to typically match or beat semantic recall at a fraction of the cost; that is the normal outcome whenever this signal fires.

Worked example, multi-hop support tickets

Questions like "did the v2.3 latency fix also resolve the memory leak from the March thread" span documents. The granularity signal fires: embed 128-token child chunks so the exact fix text matches precisely, then attach each child to its parent thread so generation reads the whole story. That is when to use parent-child chunking for RAG, and the sentence-window node parser is an off-the-shelf implementation of the retrieve-small, read-big pattern.

Token-Cost Math for Every Path

The chunk size and overlap trade-off is usually framed as recall versus precision, with a missing third axis, cost. Overlap raises the probability that a relevant span survives intact inside at least one chunk, a real benefit for boundary-straddling answers. It also inflates the index unconditionally. At 15 percent overlap on 512-token chunks, the stride shrinks from 512 to 435 tokens, so a 1M-token corpus produces about 2,299 chunks instead of 1,954, roughly 18 percent more vectors, storage, and embedding spend. So does chunk overlap improve retrieval recall? Sometimes, modestly, for answers that sit on boundaries, and your eval will show whether yours are those answers. The cost side is not conditional.

Per path on the same 1M-token corpus:

• Fixed or recursive, no overlap: about 1,954 vectors at 512 tokens, one embedding pass.
• Same with 15 percent overlap: about 2,299 vectors. The cost of chunk overlap in vector index size scales with the overlap fraction.
• Semantic chunking: one embedding pass at sentence granularity to decide boundaries, on the order of 45,000 sentence embeddings at roughly 22 tokens per sentence, plus one pass per final chunk. Roughly double the embedding work, with per-request overhead on the sentence pass.
• Parent-child: 7,813 child vectors at 128 tokens, plus parents stored for generation. Stored tokens roughly double; embedding cost stays at child level.
• Late chunking: one embedding call per full document covering all of its tokens, then pooling into chunk vectors. Total embedded tokens equal the corpus once, but each request is a whole document.
• Contextual retrieval: one LLM call per chunk at index time, about 2,300 calls for the overlap example, plus 50 to 100 blurb tokens added to every embedded chunk.

Among the strategies that clear your recall bar, the cheapest one wins. Cost belongs inside the decision, not in a post-mortem.

The half of the bill people forget is retrieval. Every retrieved token is an input token on every query, usually priced well above embedding. Top-8 retrieval at 512-token chunks feeds about 4,100 tokens per query before generation; top-5 child chunks at 128 tokens feed 640. At 10,000 queries per day, those two designs differ by tens of millions of input tokens daily.

When to Skip Re-Chunking Entirely

Late chunking versus semantic chunking is a category error people make when comparing lists: one changes what the embedder sees, the other changes where you cut. Two conditions make re-chunking the wrong move outright.

Long documents plus a long-context embedder. Take 20-page contracts at roughly 6,000 to 8,000 tokens each. With an 8k-context embedding model, late chunking embeds each contract whole and then pools token embeddings into 256-token chunk vectors, an approach introduced by Jina AI researchers and formalized in an arXiv paper. Every pooled vector has seen the full document, so an entity defined on page 2 stays meaningful in the chunk from page 14, exactly the context that naive splitting destroys, and every splitter decision disappears. The gates are real: the embedder must accept the full document, you need access to token-level pooling, and you pay for full-document embedding.

Chunks that lack their own context. Contextual retrieval prepends a short LLM-written blurb to each chunk, something like "section 3.2 of the 2024 pricing policy, covering enterprise overrides," before embedding and indexing. Anthropic's published experiment reported substantial reductions in retrieval failure rates, with further gains when contextual chunks were combined with BM25 hybrid search and reranking. The price is one LLM call per chunk at index time plus marginally longer embeddings. If your eval shows recall failing because retrieved chunks are context-free islands rather than wrong-sized, chunk headers fix that without touching the splitter.

The third skip condition is the simplest. If the baseline already clears your bar, say recall@10 above 0.9 on gold sources, stop chunking. Reranking and query rewriting usually move recall more than another splitter will.

The 90-Minute Runbook

1. 0 to 30, build the eval. Fifty real questions, one gold source each, the scorer above.
2. 30 to 45, baseline. Recursive splitting at a mid-size config, reranking off. Log recall@10, precision@10, vector count.
3. 45 to 60, read the signals. Structure density, the 30-query granularity split, embedder window. Each is a script or a pivot table.
4. 60 to 80, test one alternate. The signals indicate at most one branch: structure-aware, semantic, parent-child, late chunking, or contextual headers. Run it under identical controls.
5. 80 to 90, decide and log. Compare recall@10 and cost per query, ship the cheapest strategy that clears the bar, and write the decision log: corpus snapshot identifier, chunking parameters, scores, date.

Re-test when the corpus doubles or drifts into new domains, when the query mix tilts toward a different granularity, or when you swap embedding models. The log turns opinion into system: your RAG chunking strategy becomes a measurement with a history, and the next person who asks why 512 gets a link to numbers instead of a shrug.