AWS Cuts RAG Token Costs 33% With a Two-Call Compression Pattern

Retrieval-augmented generation pipelines on Amazon Bedrock have a well-known cost leak: high-recall retrieval returns 5–20 chunks per query to maximise the chance of including relevant content, and all of those tokens flow into the primary model on every call. AWS engineers Aakanksha Veesam and Amit Maindola published a post-retrieval compression pattern on 21 August 2026 that addresses this directly — routing retrieved chunks through a smaller, cheaper model before they reach the expensive answer model. The benchmark results are concrete enough to warrant serious attention from anyone running RAG at support or enterprise scale.

The Two-Call Architecture

The pattern inserts a single AWS Lambda function between retrieval and the final model call. The function makes two sequential calls through the Amazon Bedrock Converse API: first to a compression model (Anthropic Claude Haiku), then to the primary answer model (Anthropic Claude Sonnet). The retriever — which can be an Amazon Bedrock Knowledge Base backed by Amazon OpenSearch Serverless — returns the full top-k chunks to the Lambda function unchanged. Claude Haiku receives those chunks alongside the user query and outputs only verbatim spans directly relevant to the question, preserving [CHUNK_ID: <id>] markers so downstream citation remains accurate. Claude Sonnet then receives that compressed evidence set rather than the raw retrieval output.

The compression call runs at temperature=0.0 to keep extraction deterministic. The system prompt forbids paraphrasing or summarisation and instructs the model to emit NO_RELEVANT_EVIDENCE for chunks containing no useful signal — a design choice that prevents the smaller model from hallucinating connective tissue between spans. The Lambda function reads both model IDs from environment variables and initialises the Bedrock Runtime client with adaptive retries. This maps well to how teams think about prompt optimisation for production LLM pipelines, where the real leverage is in what you send to the expensive model, not just how you phrase it.

Benchmark Results Across 500 Questions and 500,000 Documents

The evaluation used a corpus of more than 500,000 documents spanning nine enterprise source types — chat messages, email, issue-tracker tickets, shared-drive documents, CRM records, meeting transcripts, code repositories, and wiki pages — and a question set of 500 queries across 10 categories. Answers were scored by an LLM judge across four dimensions: correctness, completeness, citation accuracy, and conciseness. Hallucination rate was tracked separately as the share of answers containing at least one claim unsupported by the reference.

Compression alone achieves an 8.6× reduction in tokens reaching the primary model, delivering a 33% cost reduction while holding composite quality at 97.5% of baseline. Adding Bedrock's Rerank API before compression pushes the token reduction to 10.1× and cost savings to 36%, with quality at 97.6%. Correctness stays within 0.07 of baseline across both optimised conditions. The hallucination reduction — 7 points under compression alone and 13 points with reranking added — is structural: the primary model has less irrelevant text to confabulate over. Cost savings also drop by query difficulty: from 37% on typical queries to 26% on hard queries under compression alone, and from 40% to 30% with rerank plus compression, which sets honest expectations for workloads heavy on multi-hop analytical questions.

The Economics and Where They Break Down

For a retrieved context of R tokens, a compression ratio of c, a final answer output of A tokens, and per-token prices for both models, the baseline pays R × P_large_in + A × P_large_out. With compression, the bill becomes R × P_small_in + (R/c) × P_small_out + (R/c) × P_large_in + A × P_large_out. The pattern is net-positive when retrieved context is large, the price ratio between large and small models is high, and a meaningful fraction of retrieved content is irrelevant to any given query.

AWS identifies four workload archetypes where the fit is strongest: regulated-industry policy assistants retrieving 8,000–15,000 token bulletins for single-clause queries; customer-support copilots running hundreds of thousands of tickets per month; internal engineering assistants over verbose runbooks and wikis; and financial research pipelines over dense 10-K sections and earnings transcripts. The pattern is flagged as a poor fit for sub-second conversational chat with small retrieved context — in that case, prompt caching and Bedrock Intelligent Prompt Routing are recommended instead. For qualifying workloads, the authors suggest a feature-flag rollout with an evaluation set drawn from de-identified production logs, running both pipelines against the same retrieval results so compression is the only variable.

The broader point is that post-retrieval processing — sitting between the vector index and the frontier model — is becoming its own optimisation layer. The same economic logic that makes small language models attractive as general-purpose components applies with particular force to filtering tasks, where the smaller model never needs to reason, only to extract. As primary model input pricing remains the dominant cost lever in managed RAG services, patterns that reduce that token count without restructuring retrieval or retraining anything will see rapid adoption across enterprise deployments.