Chunking Strategies for RAG (2026): Fixed-Size vs Semantic vs Hierarchical, Benchmarked

Most RAG tutorials treat chunking as a two-line setup step: instantiate a text splitter, call split_documents, move on. This is understandable — the step is fast, the code is simple, and the pipeline keeps moving. What gets lost is that chunking defines the atomic unit of retrieval. Every downstream component — the embedding model, the vector index, the LLM context window — operates on whatever chunks you created. If a chunk cuts a concept in half, no embedding model can fix it. If a chunk runs 2,000 tokens but your embedding model was trained on 512-token sequences, the tail of the chunk will be underweighted. These are not edge cases; they are systematic biases that degrade every query against that index.

The practical consequence is that chunking strategy interacts with corpus structure in ways that generalize poorly across domains. A 512-token fixed-size split works reasonably well on structured technical documentation where paragraphs happen to be short and self-contained. The same split is damaging on legal contracts where a defined term introduced in paragraph 3 is referenced throughout a 40-page document and splitting severs that dependency. Semantic chunking, which detects topic shift points using embedding similarity, can recover some of that structure — but at significant compute cost and with its own failure modes on highly repetitive corpora like dense FAQ pages.

Understanding this interaction is the core skill this article tries to develop. The benchmark numbers matter less than the mental model: chunk size and strategy are not global settings you optimize once; they are corpus-specific choices you make deliberately and then validate empirically. Every engineering team that runs a recall@10 evaluation against their actual corpus before deploying discovers something they did not expect.

Fixed-size chunking splits a document into contiguous windows of exactly N tokens, with an optional overlap of M tokens between consecutive chunks. It requires no ML, no linguistic analysis, and no corpus-specific configuration beyond N and M. It is deterministic, fast, parallelizable, and easy to reason about. For these reasons it is the right starting point for any new RAG project: build the baseline first, then measure whether a more complex strategy actually improves recall on your data.

The performance ceiling is the strategy's main limitation. On technical documentation in the BEIR-adjacent evaluations we surveyed, fixed-size 512-token chunks achieved recall@10 around 61%. On customer support Q&A corpora, where answers are often shorter than 512 tokens and a fixed window frequently includes part of a neighboring Q&A pair, recall dropped to around 58%. Legal contracts fared worst at approximately 52%, because critical definitions and cross-references span sections in ways that token counting has no way to detect.

The practical recommendation is to treat fixed-size as your first checkpoint, not your destination. Run it, measure recall@10 on a held-out set of 50-200 query/answer pairs, record the number, then try the next strategy. If your corpus happens to be pre-segmented into short, self-contained units (a FAQ where each entry is already a separate document, a product catalog where each row is independent), fixed-size may be the best strategy precisely because the corpus structure already does the semantic work. If you are working with dense long-form text, plan to iterate.

The recursive character splitter, popularized by LangChain's RecursiveCharacterTextSplitter, applies a prioritized hierarchy of separators: paragraph break, newline, space, then individual character as a last resort. It tries each separator in order and only falls back to the next when the current separator would produce a chunk larger than the target size. The result is chunks that tend to respect natural document structure — paragraphs stay together, sentences are not split mid-thought — without requiring any ML inference.

In practice this strategy outperforms fixed-size on nearly every corpus type. The evaluations surveyed show recall@10 of approximately 68% on technical docs, 64% on support Q&A, and 56% on legal contracts when using a 512-token target with a 50-token overlap. The improvement over fixed-size is largest on documentation corpora that use consistent paragraph formatting, where the paragraph separator does most of the work. The gap over fixed-size narrows on corpora with inconsistent whitespace or raw OCR output where the hierarchy of separators loses its signal.

The code to implement this in LangChain is straightforward: `from langchain.text_splitter import RecursiveCharacterTextSplitter; splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50, separators=[\"\\n\\n\", \"\\n\", \" \", \"\"])`. For code corpora, LangChain provides language-aware variants (RecursiveCharacterTextSplitter.from_language) that add function and class boundary separators. This is almost always the right choice for mixed Python/documentation repositories. The pragmatic advice is: start with recursive character splitting at 512 tokens and 50-token overlap, establish your recall@10 baseline, and only move to a more expensive strategy if you have measured a specific gap you need to close. Most production RAG systems that are performing well enough are running some variant of this strategy.

Semantic chunking replaces token counting with embedding-based topic detection. The algorithm embeds consecutive sentences or short windows, computes cosine similarity between adjacent windows in a sliding pass, and identifies points where similarity drops sharply as topic shift boundaries. The document is then split at those boundaries rather than at fixed token counts. The intuition is correct and the results reflect it: semantic chunking achieves recall@10 of approximately 71% on technical docs, 69% on support Q&A, and 62% on legal contracts in the evaluations surveyed — the highest single-strategy numbers across all three corpus types.

The cost is real. Semantic chunking requires an embedding pass over the entire corpus at index time, which adds latency proportional to corpus size and inference cost. For a 10,000-document corpus this is a one-time cost that may be acceptable; for a corpus that is updated in real time with low-latency requirements, the embedding dependency on the indexing path becomes an operational concern. The strategy also has failure modes on corpora with low semantic variance — dense FAQ pages where every paragraph discusses the same topic, for example — where the similarity threshold produces chunks that are either too large (no splits detected) or too granular (every sentence splits because topics are all slightly different). Threshold tuning is required, and optimal thresholds are corpus-specific.

A minimal Python implementation using sentence-transformers looks like this: embed all sentences with a model such as all-MiniLM-L6-v2, compute cosine similarity between sentence i and sentence i+1, identify indices where similarity falls below a threshold (typically 0.4-0.6, tuned on your corpus), and split at those indices. `from sentence_transformers import SentenceTransformer, util; model = SentenceTransformer('all-MiniLM-L6-v2'); embeddings = model.encode(sentences); sims = [util.cos_sim(embeddings[i], embeddings[i+1]).item() for i in range(len(embeddings)-1)]; split_points = [i+1 for i, s in enumerate(sims) if s < threshold]`. The threshold is the critical hyperparameter: too low and you get huge chunks; too high and you fragment every paragraph. For corpora you have not analyzed before, start at 0.5 and tune. See the build RAG with Pinecone tutorial for a worked example of this implementation in a full pipeline.

Sliding window chunking is fixed-size chunking with a deliberately chosen overlap between consecutive chunks. A chunk of 512 tokens with 25% overlap means chunk 2 starts 384 tokens into chunk 1, so the last 128 tokens of chunk 1 are repeated at the start of chunk 2. The motivation is straightforward: if a relevant passage straddles a chunk boundary, at least one of the two overlapping chunks will contain it intact. This directly addresses the boundary problem that makes fixed-size chunking underperform on corpora with dense cross-sentence dependencies.

The performance uplift is real but modest. Recall@10 on technical docs reaches approximately 67% at 25% overlap, compared to 61% for no-overlap fixed-size. On legal contracts, where long-range dependencies span more than a single boundary, the improvement is more pronounced: approximately 59% versus 52%. The cost is deterministic: 25% overlap increases the number of vectors by approximately 1.2x, with proportional increases in storage and query latency as the vector index grows. This is not free, but it is predictable and easy to budget.

The practical guidance is to use sliding window as an upgrade over fixed-size when you cannot afford the compute cost of semantic chunking but have measured that fixed-size is losing recall at chunk boundaries. The 10-25% overlap range covers most production use cases. Below 10%, the boundary coverage is minimal. Above 25%, storage costs accumulate quickly and retrieval may start surfacing near-duplicate chunks that confuse the LLM with slightly varying versions of the same passage. Some production systems add a deduplication step that drops chunks whose embedding cosine similarity to an existing chunk exceeds 0.98, which partially mitigates this at the cost of additional indexing complexity.

The fundamental tension in RAG chunking is that precision and context pull in opposite directions. Small chunks retrieve precisely — a 128-token chunk that exactly matches a query term will score high — but return too little context for the LLM to generate a complete answer. Large chunks provide rich context but their embeddings average over many topics, reducing the signal-to-noise ratio for any specific query. Hierarchical parent-child indexing resolves this tension by separating the retrieval unit from the context unit: index small child chunks (typically 128-256 tokens) for precise retrieval, but when a child chunk is retrieved, return its parent chunk (typically 512-1024 tokens) as the LLM context.

The results are the best single-strategy numbers across all three corpus types: approximately 73% recall@10 on technical docs, 70% on support Q&A, and 64% on legal contracts. The 64% on legal still trails document-level Cohere embedding at 67%, which points to the fundamental limitation of any chunk-based approach on corpora with very long-range dependencies — but for technical and support corpora, parent-child is the clear winner. The storage overhead runs 1.3-1.5x the baseline because both child and parent embeddings must be stored (though typically only child embeddings need to live in the hot vector index; parents can be retrieved from a key-value store keyed by parent ID).

Implementation requires maintaining a mapping from child chunk IDs to parent chunk IDs at index time, and a two-step retrieval: vector search returns child chunk IDs, a lookup step fetches parent text. In LangChain this is the ParentDocumentRetriever abstraction: `from langchain.retrievers import ParentDocumentRetriever; from langchain.storage import InMemoryStore; child_splitter = RecursiveCharacterTextSplitter(chunk_size=256); parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1024); retriever = ParentDocumentRetriever(vectorstore=vectorstore, docstore=InMemoryStore(), child_splitter=child_splitter, parent_splitter=parent_splitter)`. The docstore holds parent text; the vectorstore holds child embeddings only. For production deployments, replace InMemoryStore with Redis or a relational store. This architecture is more operationally complex than a flat index, but the recall uplift over recursive character splitting (approximately 5 percentage points on technical docs) is consistent enough across evaluations to justify it for teams that have established a stable corpus and are optimizing a production system. For a fresh corpus where you are still learning the data, start simpler.

A strategy absent from most chunking comparisons is no chunking at all. With the release of embedding models that accept 128k-token inputs — Cohere's embed-v4 being the most prominent as of mid-2026 — it is now technically feasible to embed entire documents and retrieve at document granularity. This eliminates all boundary problems, dependency breakage, and strategy selection overhead. For legal contracts, where the benchmark shows document-level Cohere embedding at approximately 67% recall@10 — beating every chunk-based strategy except parent-child on tech docs — this is not a theoretical option. It is a competitive approach.

The trade-offs are real and domain-specific. Long documents produce dense, averaged embeddings that can lose precision on narrow queries: asking about a specific clause in a 40-page contract will return the correct document, but the LLM still needs to locate the clause within the full document text fed as context. If your LLM context window is large enough to hold the full document (Gemini 1.5 Pro at 1M tokens, Cohere Command R+ at 128k), this works well and the document-level approach is genuinely simpler to operate. If your documents are longer than the LLM's context window, you need a second-stage retrieval step anyway, at which point you have effectively re-introduced chunking.

The practical recommendation for legal and regulatory corpora specifically is to test document-level Cohere embedding before investing in a complex hierarchical chunking pipeline. If your average document is under 50k tokens and your LLM can accept the full document, document-level embedding plus full-document context may be the highest-recall, lowest-complexity option available. For technical documentation and support Q&A, where documents are shorter but queries are more targeted, the precision loss from averaging over a full document typically makes chunk-based approaches — particularly parent-child — more effective. See Cohere vs Voyage vs OpenAI embeddings for a full comparison of model capabilities at different context lengths.

The recall@10 numbers presented in this article are drawn from public evaluations published through June 2026. Sources include the LangChain documentation benchmarks on the LangChain blog, ChromaDB's public chunking evaluation notebook, MongoDB's RAG evaluation series co-published with Voyage AI, and several academic papers evaluating BEIR datasets with RAG pipelines. Where multiple sources reported numbers for the same strategy and corpus type, we took a representative midpoint. No independent benchmarking was conducted by Digital Dashboard Hub for this article.

Several factors will shift these numbers when you apply them to your own corpus. First, the embedding model matters enormously. Most public evaluations use OpenAI text-embedding-3-small or -large, or Voyage AI's voyage-2. Numbers with a different model — especially one with a different training context window — will differ. Second, the query distribution matters. BEIR and BEIR-adjacent evaluations use specific query types; if your users ask very different questions, the ranking of strategies may change. Third, document preprocessing matters. OCR quality, HTML stripping, whitespace normalization, and header/footer removal all affect chunking quality in ways that benchmarks on clean academic datasets do not capture. Fourth, the chunk size numbers are specific to the parameter choices listed. Fixed 512 is not the same as fixed 256 or fixed 1024; the benchmark numbers would shift at different sizes.

The appropriate response to all of this uncertainty is empirical: build a held-out evaluation set of 50-200 (query, expected-answer-document) pairs representative of your actual user queries, implement two or three candidate strategies, compute recall@10 on your corpus, and make the strategy decision from your own numbers. The table and verdicts in this article are a prior to update, not a conclusion to accept. If you have not yet run a recall evaluation on your corpus, the RAG cost calculator can help you estimate the cost of running that evaluation at different corpus and query set sizes.

For technical documentation — API references, engineering wikis, product manuals — the benchmarks point to parent-child (256 child / 1024 parent) as the highest-recall strategy, with recursive character 512+50 as the pragmatic alternative when operational simplicity matters more than the last 5 percentage points. Technical documentation tends to have consistent paragraph structure that recursive character splitting handles well, and the query patterns (specific function names, configuration parameters, error codes) benefit from precise small-chunk retrieval. The parent-child architecture earns its complexity here.

For customer support Q&A corpora — helpdesk tickets, FAQ databases, community forum threads — semantic chunking performs best at approximately 69% recall@10, with parent-child close behind at 70%. The distinction matters: semantic chunking performs well here because support corpora have highly variable chunk lengths (some answers are two sentences, others are 800 words) and forcing a fixed token window either splits short answers across chunks or combines multiple unrelated answers into one chunk. Semantic boundary detection avoids both failure modes. If semantic chunking's compute cost is prohibitive, recursive character splitting at 512+50 is a reasonable fallback at 64%.

For legal contracts, regulatory filings, and compliance documents, document-level Cohere embedding at 128k context achieves the highest recall at approximately 67%, assuming documents fit within the context window and your LLM can accept full-document input. Where that is not the case, semantic chunking with overlap is the strongest chunk-based option at approximately 62%. Legal text's dense cross-referencing structure benefits from the longest possible context units, and any strategy that breaks documents into sub-512-token pieces loses critical definitional context. For code corpora, no benchmark is presented, but the consistent finding in the literature is that recursive character splitting with language-aware splitters (splitting on function and class definitions) outperforms generic token-based approaches. General English corpora — news articles, blog posts, Wikipedia — follow the pragmatic default: recursive 512+50 performs reliably and the complexity of more advanced strategies rarely yields measurable recall gains.

The benchmarks cited in this article draw from: LangChain's RAG from Scratch series and text splitter evaluation notebooks (langchain-ai.github.io, 2024-2025); ChromaDB's chunking evaluation repository (trychroma.com/blog, 2024); MongoDB and Voyage AI's RAG evaluation blog series (mongodb.com/developer, 2024-2025); the BEIR benchmark paper (Thakur et al., NeurIPS 2021) and subsequent RAG-specific evaluations that apply BEIR-style methodology; Cohere's embed-v4 technical documentation and evaluation results (cohere.com, 2025-2026). All numbers should be treated as indicative given the differences in embedding models, preprocessing, and query distributions used across these sources.

For implementation depth beyond what this article covers, the LangChain text splitter documentation provides the most complete reference for the recursive character and parent-child implementations. The sentence-transformers library documentation covers the embedding-based semantic chunking approach. For evaluation methodology, the BEIR paper itself is the best starting point for understanding what recall@10 measures and what its limitations are as a single-number summary of retrieval quality.

Related reading on this site: when RAG fails and fixes covers the seven root causes of RAG underperformance beyond chunking; build RAG with Pinecone provides a complete implementation walkthrough; RAG architecture decision tree 2026 covers the upstream decision of when to use RAG versus fine-tuning versus long-context prompting; and embedding model leaderboard 2026 covers the model selection decision that interacts directly with chunk size.