The Chunking Problem

If your retrieval pipeline is performing badly, the temptation is to blame the embedding model. Maybe try Cohere instead of OpenAI. Maybe the new Voyage release. Maybe fine-tune your own. None of this will help much, because the embedding model isn’t usually what’s broken. The chunking is.

How you split a document into pieces before embedding determines what can ever be retrieved. Get the split wrong and no amount of clever query-time work will save you – the answer simply doesn’t exist in your index as a coherent unit. Get it right and a mediocre model will outperform a state-of-the-art one fed bad chunks. This is the least glamorous part of any RAG system and the one that actually moves the needle.

Why fixed-size chunking is a problem

The default approach in every RAG tutorial – “give me 512-token chunks with 50 tokens of overlap” – is what almost nobody should actually run in production. It splits mid-sentence. It splits mid-table. It severs a heading from the paragraph that explains it. It cuts a numbered list off from the introductory clause that made the list make sense. Every one of these is a retrieval failure waiting to happen.

Concretely: imagine a chunk that ends “…there are three reasons:” and the next chunk starts “First, the regulatory framework requires…” The embeddings for those two chunks will land in completely different parts of the vector space. A query about “regulatory compliance reasons” might pull one, the other, or neither. It will almost certainly not pull both together in a way the LLM can reason over. The compression-aware framing from the previous post applies: a chunk that lacks the introductory context has thrown away the bit that mattered.

And it’s worse than just losing information. Bad chunks also pollute retrieval. If your top-5 contains three chunks that look topically relevant but are actually fragments stripped of context, the LLM will confidently generate from incomplete information. Hallucination rates on RAG systems track chunking quality more tightly than they track model size.

What structure preserves

Most documents have structure the chunker can lean on. Headings demarcate topics. Paragraphs are usually coherent units. Tables are atomic. Code blocks are atomic. Numbered lists are coherent. Most failures come from a chunker that’s blind to all of this and just counts tokens.

A respectable strategy in 2026 looks something like:

• Parse the document structurally first – Markdown headings, HTML elements, PDF layout via something like unstructured or pymupdf4llm.
• Chunk at paragraph or section boundaries, never at arbitrary token counts.
• Keep tables and code blocks whole, even if they overflow your nominal size budget.
• Prepend each chunk with its heading hierarchy. This is the part most pipelines miss.
• Where chunks are still ambiguous, prepend a one-sentence LLM-generated context summary at indexing time.

The heading prefix is free wins. A chunk becomes:

# Annual Report 2024
## Risk Factors
### Regulatory Risk
The FDA approval pathway for biosimilars introduces uncertainty
in both timing and probability of commercial launch...

Now the embedding carries the document structure with it. A query about “regulatory risk in biosimilar approval” lands in the same neighbourhood as the chunk, instead of having to overcome the chunk’s lack of context.

Anthropic’s contextual trick

Anthropic published the obvious-in-retrospect idea a couple of years back: before embedding a chunk, ask a cheap LLM to write a one-sentence description of how the chunk fits into the larger document, and prepend that. Something like:

This chunk is from a 2024 financial filing,
in the Risk Factors section about regulatory risk,
summarising the FDA approval pathway for biosimilars.
[original chunk text follows]

The embedding now carries document-level context, not just the local text. The cost is one cheap LLM call per chunk at index time – a one-off, not a per-query cost. Most teams who try it see retrieval precision improve more than from any model swap I’ve seen them attempt. Anthropic reported around 35% reduction in retrieval failure when they first wrote it up. The numbers will be smaller on better-structured corpora and larger on worse ones, but the direction is the same.

Semantic chunking and its limits

A more exotic option is semantic chunking, where you embed sentences and look for break-points by detecting where the embeddings shift meaningfully. The idea is appealing – chunks should follow the natural seams in meaning, not arbitrary token counts.

In practice, I’ve found it slow, expensive at index time, and not noticeably better than structural chunking on most corpora. If your documents are heavily structured (financial filings, regulatory text, technical docs), the structure is doing more work than the embeddings will. If your documents are unstructured stream-of-consciousness (call transcripts, chat logs), semantic chunking has more to offer, but you’re probably better off pre-processing into structured form before retrieval anyway.

What nobody benchmarks

Here’s the awkward truth. Every RAG benchmark you’ve read evaluates retrieval quality assuming the chunks are given. None of them benchmark the chunking step itself, because doing that would require ground-truth annotation of what a good chunk looks like for the specific corpus, and that’s expensive and corpus-dependent. So the literature trains your intuition on the wrong layer of the stack.

If you do nothing else for retrieval quality this quarter, build a small evaluation set: fifty questions where you know the answer is in a specific section of a specific document. Then check whether your current chunking lets that section even be retrieved as a coherent chunk. The first time most teams do this, the answer is “no, the relevant content has been carved up across three chunks, none of which is sufficient alone.” Everything you build on top of that pipeline – rerankers, query rewriting, agentic orchestration – is downstream of that one failure.

The boring conclusion

Chunking is unglamorous. It doesn’t have a leaderboard. The people who write tutorials default to fixed-token chunks because that’s what fits in a tutorial. None of which makes the problem go away.

If your retrieval is bad, look there first. Look at it second too. Most of the leverage in a RAG pipeline lives in document parsing and chunking, not in the parts that get the marketing budget.