Six Cheaper Rungs Under RAG: When Not to Reach for the LLM
In this article
Defaulting to an LLM call is the fastest way to ship a wrong solution. That is the core argument Kezhan Shi lays out in a Towards Data Science series on enterprise document intelligence: RAG answers one kind of question, and the reflex to reach for it obscures six cheaper rungs that, in most production workloads, settle requests faster, at lower cost, and with an audit trail the LLM call cannot produce.
The framing the series uses is a ladder. Exact match sits at the bottom — when the request carries a clean identifier, character comparison resolves it in milliseconds. Spell correction handles the next rung: one transposed letter should not trigger a generation call when SymSpell and a corpus-derived vocabulary handle it deterministically. Keyword search over an expert-curated vocabulary sits above that, and embeddings cover the vocabulary gaps that structured lookup misses. Only at the top of the ladder does an LLM call earn its place, and the series argues that most enterprise requests never climb that high.
The Spelling and OCR Problem Embeddings Only Partially Solve
Article B01 maps three distinct noise sources onto the pipeline: user typos, transcription artifacts from fast input, and OCR character errors in scanned documents. These three are not equivalent. Forty years of classical spell-correction techniques — Levenshtein distance, BK-trees, Soundex phonetic matching, and SymSpell — handle the user-typo case well. Embeddings and LLMs absorb the other two. The practical split the series defends: apply spell correction against the corpus vocabulary at question-parse time, leave the document corpus itself noisy, and design retrieval to tolerate that noise.
The local-embeddings sub-series (B11) makes a pointed empirical claim: on OCR-noisy text, one locally hosted Ollama embedder produced better chunk separation between answer-bearing and decoy passages than the cloud-hosted text-embedding-ada-002 reference. Clean-text quality still favors the cloud model; residency and a specific retrieval-noise band favor the local one. That finding connects directly to the broader argument about pipeline architecture driving AI gains rather than model upgrades alone.
Table Extraction: Why Four Representation Levels Replace One Linear Decision
Article B04 identifies tables in PDFs as the site where most RAG pipelines fail silently. A linear decision rule collapses because the relevant dimensions are orthogonal: a table can simultaneously be column-typed and large, or mixed-row and deeply nested, and those combinations do not map onto a single ordered policy.
The series proposes four representation levels: rows stored as lines in line_df, a separate table_df, columnar storage with named and typed columns, and columnar storage for heterogeneous content. A per-table diagnostic on five orthogonal axes determines the level, and a set of idempotent operations moves tables between levels when the initial assignment is wrong. Most tables stay at the simplest level and only pay escalation cost when the diagnostic demands it — a principle of architectural specificity over brute-force processing.
Local LLM Sizing: What the Eleven-Model Sweep Found
The local-LLM sub-series (B10, B11, B12) answers the self-hosting question from first principles. Article B12 pulls eleven Ollama models locally, ranging from 815 MB to 9.1 GB on disk. JSON structural validity holds from the 1B-parameter class upward. Literal extraction without fabrication — the property that matters most for audited document pipelines — does not become reliable until 7B parameters. The smallest model the series designates as production-ready is qwen2.5:7b.
Article B10 adds a specific failure mode: qwen3:4b, a smaller and architecturally newer model than qwen2.5:7b, silently drops the output schema during the LLM-confirm stage the pipeline uses for retrieval validation. A smaller parameter count combined with a reasoning-optimized architecture is not a drop-in replacement for a generation-optimized model of the same nominal class.
| Technique | Problem class | Audit trail | Noise tolerance | Series coverage |
|---|---|---|---|---|
| Exact match | Clean identifier present | Full — rule named | None | Ladder rung 1 |
| SymSpell / BK-tree spell correction | User typos, one-character errors | Full — correction rule logged | User input only | B01 |
| Expert-vocabulary keyword search | Domain-bounded free text | Full — matched term logged | Low | Ladder rung 3 |
| Dense embeddings (cloud) | Vocabulary gaps, paraphrase | Partial — similarity score | Moderate; degrades on OCR noise | B11 |
| Local Ollama embeddings | Paraphrase under residency constraint | Partial — similarity score | Better than cloud on OCR-noisy retrieval | B11 |
| LLM generation (cloud or local ≥7B) | Cross-reference, reasoning, synthesis | Weakest — prompt-dependent | High | B05, B10, B12 |
Benchmarking Discipline Over Model Benchmarks
The thirteen-model benchmark in B05 — covering OpenAI tiers, Anthropic models, self-hosted Mistral, Llama, Phi, and Qwen — reports per question type rather than aggregate accuracy, because aggregate scores obscure failure modes. Point lookups, list retrievals, and cross-references each expose different model weaknesses. The series finds that a strong dispatcher narrows the performance gap between a $20-per-million-token model and a free self-hosted alternative more than the model choice itself does — echoing the systems-engineering gains that now rival raw scaling.
The most expensive tool in the NLP stack is also the one that explains itself least. Building document pipelines that reserve LLM calls for cases that genuinely require them — routing everything else through deterministic, auditable, cheaper methods — is the architecture that survives a compliance audit, a latency SLA, and a corpus that cannot leave the building.