Fixed-size, semantic, recursive, structure-aware, and LLM-driven chunking—trade-offs and when to test.
Almost done! Once the most relevant chunks are re-ranked, they are fed into the LLM. This model combines the user's original query with the retrieved chunks in a prompt template to generate a response that synthesizes information from the selected documents. This is depicted below: The LLM leverages the context provided by the chunks to generate a coherent and contextually relevant answer that directly addresses the user’s query. Since chunking is the very first step in any RAG pipeline, it’s important to understand the different ways it can be done. 5 chunking strategies for RAG Here’s the typical workflow of RAG:
Since the additional document(s) can be large, step 1 also involves chunking, wherein a large document is divided into smaller/manageable pieces. This step is crucial since it ensures the text fits the input size of the embedding model. Here are five chunking strategies for RAG: Let’s understand them!
1) Fixed-size chunking
Split the text into uniform segments based on a pre-defined number of characters, words, or tokens. Since a direct split can disrupt the semantic flow, it is recommended to maintain some overlap between two consecutive chunks (the blue part above). This is simple to implement. Also, since all chunks are of equal size, it simplifies batch processing. But this usually breaks sentences (or ideas) in between. Thus, important information will likely get distributed between chunks.
2) Semantic chunking
Segment the document based on meaningful units like sentences, paragraphs, or thematic sections. Next, create embeddings for each segment. Let’s say we start with the first segment and its embedding. If the first segment’s embedding has a high cosine similarity with that of the second segment, both segments form a chunk.
This continues until cosine similarity drops significantly. The moment it does, we start a new chunk and repeat. Here’s what the output could look like: Unlike fixed-size chunks, this maintains the natural flow of language and preserves complete ideas. Since each chunk is richer, it improves the retrieval accuracy, which, in turn, produces more coherent and relevant responses by the LLM. A minor problem is that it depends on a threshold to determine if cosine similarity has dropped significantly, which can vary from document to document.
3) Recursive chunking
First, chunk based on inherent separators like paragraphs, or sections. Next, split each chunk into smaller chunks if the size exceeds a pre-defined chunk size limit. If, however, the chunk fits the chunk-size limit, no further splitting is done. Here’s what the output could look like:
As shown above: First, we define two chunks (the two paragraphs in purple). Next, paragraph 1 is further split into smaller chunks. Unlike fixed-size chunks, this approach also maintains the natural flow of language and preserves complete ideas. However, there is some extra overhead in terms of implementation and computational complexity.
4) Document structure-based chunking
It utilizes the inherent structure of documents, like headings, sections, or paragraphs, to define chunk boundaries. This way, it maintains structural integrity by aligning with the document’s logical sections. Here’s what the output could look like:
That said, this approach assumes that the document has a clear structure, which may not be true. Also, chunks may vary in length, possibly exceeding model token limits. You can try merging it with recursive splitting.
5) LLM-based chunking
Prompt the LLM to generate semantically isolated and meaningful chunks. This method ensures high semantic accuracy since the LLM can understand context and meaning beyond simple heuristics (used in the above four approaches). But this is the most computationally demanding chunking technique of all five techniques discussed here.
Also, since LLMs typically have a limited context window, that is something to be taken care of. Each technique has its own advantages and trade-offs. We have observed that semantic chunking works pretty well in many cases, but again, you need to test. The choice will depend on the nature of your content, the capabilities of the embedding model, computational resources, etc.