Row-Level RAG Chunks Cut Table Context by 7.7x
In this article
Paragraph-level and page-level chunking share a common failure: when the answer to a user's question is a single row inside a multi-row table, the retriever sends the whole rectangle to the generation model and leaves it to filter. Kezhan Shi's August 2026 piece in Towards Data Science, part of the Enterprise Document Intelligence series, fixes this by building a second retrieval index directly on top of the existing parsed document frame — one that turns each table body row into its own retrievable chunk, column headers included.
The payoff is concrete. On a slice of an insurance guarantees table, a targeted lookup returns a single 122-character row where the full slice spans 943 characters — a 7.7× context reduction. On Table 1 of Attention Is All You Need (Vaswani et al., arXiv:1706.03762, 2017), the same move sends 124 characters to the generation model instead of the 528-character full table — a 4.3× ratio on a compact four-row example. The ratio tracks row count directly, so a 40-row contract produces roughly a 40× saving on a targeted query.
This sits within the broader pattern that pipeline architecture, not better models, drives retrieval quality gains: the generation model is being asked to filter rows that the retriever should have excluded before the context window was filled.
The serialize_table_rows Primitive
The implementation reads only line_df, the line-level DataFrame that both Docling and Azure Document Intelligence already emit. Both parsers render tables as markdown-pipe lines, giving a shared detection contract: a pipe row starts and ends with | with at least one interior pipe, a separator row of dashes and colons marks the header/body boundary, and consecutive pipe rows on the same page belong to the same table. A non-pipe line resets the group.
The serialize_table_rows function scans these groups, reads the header from the row immediately above the separator, and emits one output row per body line. Each chunk takes the form col: val | col: val | … — a format an LLM reads as naturally as prose and that existing keyword or embedding retrievers can index without modification.
The returned DataFrame is a second index, not a mutation of line_df. Each row-level chunk is keyed on (page_num, line_num), which slots into the citation contract from the broader series: when the generation model quotes a specific cell value, attribution resolves to an exact span in the source document.
Dispatcher Logic and Scale Selection
The row-level index adds a second scale the dispatcher selects by question shape. Three question types cover the practical space:
| Query Shape | Example | Dispatcher Routing | Generation Input |
|---|---|---|---|
| Targeted lookup | "What is the cap for vehicle theft?" | Row-level index | One row + column headers |
| Synthesis | "Which events are covered?" | Row-level fires; widening rule triggers when k / n_body_rows ≥ 0.6 | Whole table (no regression vs. table-level retriever) |
| Mixed | "Compare the vehicle-theft cap with the fire cap" | Row-level; two rows stitched | Two rows, lower token cost than full-table dump |
The widening threshold of k / n_body_rows ≥ 0.6 is explicit in Shi's design: below it, returning the row-level answer is the honest move; above it, claiming a synthesis from 2 of 40 matching rows would be a hallucination the retrieval layer should preempt. The use_row_level: bool flag on the retriever sits alongside existing use_toc, use_keywords, and use_dense flags and is set by the question parser, so the paragraph-level path runs unchanged when the flag is off.
The Multi-Row Header Edge Case
The one table shape the single-line serializer fails on silently is the spanned two-line header, common in academic and financial tables. Shi's example is Table 2 of the Attention paper (page 8), which prints BLEU and Training Cost (FLOPs) on a top header row spanning sub-columns EN-DE and EN-FR. Docling flattens this into a first pipe row with blank cells, a separator, and a body row that is actually the second header line — all labels, no digits — before the numeric rows begin. A naive serializer reads that label-only row as the first data row and misroutes keyword searches for EN-DE to it rather than to the numbers below.
The fold_multirow_header step handles this inside the serializer. When a header cell is empty, the parser has flattened a spanning cell. The function forward-fills the spanned label rightward, then checks whether the first body row is all-text with digits on the row below it. When both conditions hold, it concatenates the two header levels column by column — producing BLEU EN-DE, BLEU EN-FR, Training Cost (FLOPs) EN-DE, Training Cost (FLOPs) EN-FR — and drops the sub-header line from the body. An ordinary single-line-header table or an all-text glossary table passes through untouched.
Pipeline Wiring
row_df = serialize_table_rows(line_df) is computed once and cached alongside line_df. The retriever gains a use_row_level: bool flag; the question parser sets it by question shape. Nothing about the paragraph-level retriever changes.
The technique composes cleanly with direct corpus interaction patterns for AI agents, where retrieval precision at the sub-document level separates a useful agent response from a hallucinated one. The implementation ships in the companion repo doc-intel/notebooks-vol1 as runnable notebooks against the Attention paper's Table 1, so practitioners can validate the exact character-count ratios cited here before applying the pattern to production document corpora.