Glossary
Document chunking
Document chunking is the splitting of documents into smaller passages before they are indexed, so AI search can match a question to the right part of a document. Each chunk receives its own embedding, so where the cuts fall decides what retrieval can find and what an answer can cite.
Why do whole documents match poorly?
An embedding condenses whatever text it is given into one list of numbers. For a 60-page maintenance manual, that single summary averages everything in it and matches no particular question well. Cut into passages, each of the embeddings describes one idea, and a question about a torque setting lands near the paragraph that states it. Language models also accept a limited amount of text per request, so retrieval has to return passages rather than files. Chunking is the first preparation step of retrieval-augmented generation, and its typical failure is the right document with the wrong paragraph: the system finds the correct manual and quotes the section beside the one that holds the answer, a cut problem rather than a model problem.
Four ways to cut
| Method | Where it cuts | Works well for | Weak spot |
|---|---|---|---|
| Fixed size | Every set number of tokens, often with overlap | Uniform, unstructured text | Cuts through sentences, steps and tables |
| Structure-aware | At headings, sections and list boundaries | Manuals, policies, wikis | Sections vary widely in length |
| Topic shift | Where the subject of the text changes | Long narrative documents | Needs an extra model pass to find boundaries |
| Format-aware | Along clauses, code blocks or table rows | Contracts, source code, spreadsheets | Needs a parser for each format |
Chunk size trades precision against context
Small chunks match narrow questions closely but lose the surrounding text that gives a sentence its meaning, such as which product a limit applies to. Large chunks keep that context but blur the match, because several topics share one embedding and more irrelevant text reaches the model. Overlap, which repeats the end of one chunk at the start of the next, keeps boundary sentences whole in at least one chunk and enlarges the index. Two cuts do the most damage: a table separated from its header row, which leaves numbers without labels, and a rule split from the condition that governs it, so retrieval returns only half.
Changing the rule means rebuilding the index
A new chunking rule changes every chunk, and every chunk needs a new embedding, so the whole corpus is read and re-embedded before the vector database can be replaced. Keeping original files intact, with the source version stored beside each chunk, lets the rebuild run without hunting for origins and lets answers cite exact versions. Where those originals sit and how fast they can be re-read are questions for RAG storage. Poor cuts are also a leading upstream cause of RAG hallucination.