Almost every retrieval system starts life with a fixed-size chunker, because every tutorial starts there. It is a reasonable default for a demonstration and a poor one for a corpus, and the gap between those two facts accounts for a surprising share of disappointing pilots.
The problem is not the size. It is that a fixed window splits on token count, and token count has no relationship to where meaning ends.
What a bad split costs you
Consider a condition report where a finding reads: 'Section 4.2. Chamber 7, Site ref WTW-0412. Spalling observed to the north wall, moderate, monitor at 12 months.' A fixed window that splits between the site reference and the finding produces two chunks, neither of which is retrievable by a question that names the asset and asks about its condition.
The embedding model is not at fault, and no amount of upgrading it will recover the relationship, because the relationship is no longer present in either chunk. This is why teams who swap models and see no improvement are often looking in the wrong place.
Structure first, size second
Split on the structure the document already has: sections, clauses, conversational turns, table rows grouped by their subject, then enforce a maximum size within those boundaries rather than across them.
Attach the parent context to every chunk: document title, identifiers, revision state, section heading. It costs tokens and it is almost always worth them, because it is what makes a chunk retrievable in isolation.
- Contracts and policies: clause boundaries, with the defined-terms section attached as context
- Correspondence: conversational turns, with the thread summary and participants attached
- Reports: section boundaries, with tables extracted as structured records rather than flattened prose
- Transcripts: speaker turns grouped into topical segments, never fixed durations
Then measure it
None of the above is worth asserting without evidence, and the evidence is cheap to produce. Build a labelled question set of two to three hundred items, stratified by document family. Run each chunking strategy through the same retrieval configuration. Compare recall@k per family.
In our experience the difference between fixed-window and structure-aware chunking on a heterogeneous corpus is larger than the difference between any two current embedding models. That ordering is worth internalising, because the model upgrade is the change teams reach for first and the chunker is the one that pays.
The corollary is uncomfortable: if you do not have a labelled set, you cannot know which chunking strategy you should be using, and you are choosing by intuition on the decision that matters most.
