Kimi K3's 1M-Token Window Costs 16× More Than RAG on 12 Questions

Moonshot AI shipped Kimi K3 in July 2026 with a one-million-token context window, immediately raising a practical question for retrieval engineers: if the entire knowledge base fits inside a single prompt, is RAG still worth the operational overhead? Data scientist Sarah Schürch answered that empirically, running the same 12 questions through both a tuned RAG pipeline and a full-corpus prompt against identical model settings, then grading every response blind on three rubrics. The failure modes she uncovered along the way carry as much signal as the scores.

Experimental Design

The corpus comprised 33 articles across 32 files, totalling 127,068 tokens counted with tiktoken and cl100k_base. That volume is 12% of Kimi K3's context window. Moonshot billed 127,346 input tokens per long-context request once the system prompt and question were included.

Both paths shared an identical SYSTEM_PROMPT instructing the model to cite article titles for every fact and to decline rather than guess when information was absent. Temperature was fixed at 1 — the only value Kimi K3 permits — making exact run replication impossible.

The RAG path split the corpus into 788 chunks of 900 characters with 150 characters of overlap, embedded them with all-MiniLM-L6-v2, and retrieved the five nearest neighbours per query, producing roughly 1,200 tokens per request. The long-context path consumed 127,346 tokens per request — approximately 100× more per call. Questions were divided into three groups of four: single-fact lookups (Group A), cross-article synthesis (Group B), and corpus-wide aggregation requiring visibility of all 32 articles simultaneously (Group C).

Benchmark Results

The quality gap is concentrated entirely in completeness. RAG's grounded score matched long-context exactly at 2.00 — the system prompt's instruction to refuse rather than guess held, so on Group C questions the model cleanly admitted it could not answer from five chunks rather than fabricating. On question C1 (count of articles linking to a GitHub repository), RAG correctly stated it could not determine the total; long-context identified all 13 and listed them with supporting quotes. On C2 (most-frequently mentioned tools), RAG again declined cleanly. That behaviour is a direct consequence of the system prompt, not retrieval design — an important distinction for anyone moving RAG into production.

The Reasoning-Model Tax on Latency and Cost

On Group A single-fact questions the two paths were close; on Group B, long-context needed two to three times as long; on Group C the gap reached 273.7 seconds versus 46.3 seconds for question C3. On C2, the prefix cache fully serviced 127,232 of 127,342 input tokens — yet C2 was still the second-slowest question at 208.3 seconds. The bottleneck was thinking tokens, not context ingestion.

Kimi K3's reasoning effort defaults to max and cannot be disabled, only attenuated via the reasoning_effort parameter (low, high, max). Thinking tokens consume the same max_completion_tokens budget as the visible answer. In the first run, max_completion_tokens was set to 800; on harder questions the model spent all 800 tokens reasoning and returned empty content with finish_reason: length. Twelve of 24 answers came back empty or truncated mid-word before the bug was caught — and every failed call appeared successful across every other logged metric. The fix was to surface finish_reason explicitly and flag truncated responses at runtime.

The failed attempt on question C2 cost $0.0635 against $0.0187 for the successful retry — the model thought for 3,997 tokens the first time versus 884 the second, with identical inputs and no explanation beyond temperature=1 non-determinism. This same non-determinism complicates automated LLM prompt optimisation in production, where reproducible measurement baselines matter equally.

When Each Approach Wins

Prefix caching is the variable that makes long-context cost planning unreliable. Across 8 long-context calls, caching engaged on only 3, with no discernible pattern — and after a five-day gap the cache had expired entirely despite an unchanged corpus. The uncached price per call was $0.3916 against $0.0466 cached, roughly 8× more expensive. Pre-run projections assuming caching after the first call underestimated actual spend by approximately 30%.

The daily token quota on Moonshot's entry tier is 1,500,000 tokens; the long-context path alone requires 1,528,152 input tokens for 12 questions, breaking the quota regardless of cost.

Schürch's practical decision rule: for a corpus well below 20% of the context window receiving infrequent queries, long-context eliminates chunk tuning, embedding model selection, and vector index maintenance while delivering perfect completeness. At 12 queries the $3.59 cost delta is negligible. At 12,000 queries the same ratio produces roughly $3,800 versus $230, at which point RAG's operational complexity is financially justified. The latency ceiling — 111 seconds average, rising above 170 seconds on Group C — rules long-context out for any interactive application regardless of query volume.

Long-context windows don't eliminate retrieval engineering so much as relocate the complexity: from chunk size and embedding quality to quota management, cache reliability, and reasoning-budget allocation. The number to pin down at project start is not price per token but expected queries per knowledge base — that figure determines which complexity trade-off is actually cheaper to carry.