What chunking is: a complete guide to why where you cut decides what gets found

Chunking divides a long document into pieces sized for retrieval and generation, and where those cuts fall determines whether the passage a later question needs can actually be found.

Cut mid-sentence and the meaning disappears

Cut in the middle of a sentence and the meaning goes with it.

A document cannot be fed into retrieval whole, so it has to be divided. The simplest method counts characters and cuts at a fixed length. It is easy to implement and produces evenly sized pieces.

The difficulty is that documents are not composed by character count. Split a table down the middle and you get one chunk with the header and another with only values. Split a clause in two and the condition parts company with its exception. When retrieval returns only one of those chunks, the model builds an answer from half the information. Poor answer quality frequently originates here rather than with the model.

What chunking actually is: how structure-based differs from length-based

Length-based and structure-based chunking across three axes

First, the criterion differs. Length-based cutting uses character count. Structure-based chunking cuts on the document's logical units — sections, subsections, tables — and treats length as a secondary condition.

Second, the result differs. Length-based cutting produces even pieces with interrupted content. Structure-based chunking produces uneven pieces where each one carries a complete meaning.

Third, the prerequisite differs. Length-based cutting needs only text. Structure-based chunking needs the document's structure to have been analysed first, which means chunking quality depends on the quality of the structuring stage before it.

The two are not alternatives. Dividing by structure and then splitting only the over-long pieces by length is the practical combination.

Five decisions in the design

Five decisions to settle

First, the boundary criterion. Section or subsection, and whether tables stay whole, decided per document type.

Second, the size range. Too small and context is missing; too large and unrelated content mixes in. Production configurations commonly divide at the scale of a few thousand characters.

Third, whether to overlap. Overlapping consecutive chunks slightly compensates for sentences cut at a boundary.

Fourth, what to carry alongside. Document title, section title, page number, and effective date attached to each chunk improve both retrieval accuracy and the ability to cite grounding.

Fifth, how tables are handled. A table behaves differently from prose and can fail to retrieve if left as it is. Storing a summary sentence with it works well.

How chunking is applied in practice

Split the rules by document type

Statutes and notices with a clear clause structure, manuals where sections and subsections form a hierarchy, official correspondence that is short and self-contained, and filing forms where the table is the substance each suit a different criterion.

Processing everything with one rule guarantees a problem somewhere. Classifying document type first and applying type-specific rules is the better arrangement.

Fix the rules from retrieval failures

Chunking rules are not finished in one pass. Once real questions start arriving, failures nobody anticipated come to light.

Three patterns dominate: related content scattered across chunks with only one retrieved, information held in a table that does not match a prose question, and chunks too small to carry context. Each is addressed by revisiting boundaries, adding table summaries, and adjusting size. Collecting failure cases grows the basis for fixing the rules and the evaluation set at the same time.

Chunking in the Korean environment

Korean public documents are revised frequently, with versions divided by effective date. Carrying effective date and revision history on each chunk prevents a repealed provision from being retrieved.

Format is a condition too. Tables in Korean word processor files easily lose their merge information during conversion, and chunking a table that has already lost it produces pieces that carry no meaning. Whether the native format can be handled at the structuring stage ultimately decides chunking quality.