AKV Cuts Agent Serving Cost 27.7% Without Rewriting History

Tsinghua University and the Chinese University of Hong Kong have open-sourced AKV, an Apache-2.0 serving system for KV cache engineering in LLM agents that builds on SGLang. MarkTechPost reports the researchers’ main result: across three models and four agent benchmarks, average serving cost falls 27.7% and average task score rises 1.7 points compared with threshold summarization.

Instead of summarizing old message history, agents drop selected cache entries and reposition the rest. The text trajectory stays intact. Summarization edits the prefix and forces the retained suffix to be re-prefilled; AKV changes only which KV states remain visible to attention, skipping that recomputation. In the paper’s cost analysis, KV cache reads make up 82.4% of total API cost.

Cache operations replace text editing

Context engineering trims or summarizes text history, but editing a prefix invalidates every later KV state. The retained suffix must be re-prefilled, so harnesses summarize late and aggressively, sometimes losing middle-round details. AKV leaves the text trajectory untouched and defines three operations: Append adds new tokens normally; Drop removes chosen entries from future attention without changing positions; Repos rewrites retained keys to positions 0 through r−1 using a RoPE rotation while values stay unchanged. MarkTechPost reports that adding Repos raised GPT-OSS-120B’s BrowseComp-Plus pass rate from 55% to 62%.

Because identical tokens can produce different cache states under different operation histories, plain prefix matching is unsafe. AKV extends SGLang’s radix tree with drop and reposition nodes, frees dropped KV only when no active request still needs it, and uses segment-batch attention to restore evicted states in one prefill pass.

What the benchmark table shows

AKV was tested on GPT-OSS-120B (5.1B active parameters), MiniMax-M2.7 (230B total, 10B active), and MiroThinker-1.7-Mini (30B total, 3B active) across BrowseComp-Plus, GAIA-Text, SWE-bench Verified, and xbench-DeepSearch-2510. The main policy is a tool-response rolling drop that keeps the latest 12 tool responses. Standard summarizes at 96K tokens (192K for MiroThinker); Text Engineering summarizes at 48K. The researchers estimate costs from published API rates, with repositioned tokens charged at the cache-hit rate.

Table values are average pass rate (%) and cost (US cents per task) across the four benchmarks from Table 1 of the researchers’ paper, as reported by MarkTechPost. AKV also cuts average cost 21.9% versus Text Engineering. The best single case is GPT-OSS-120B on SWE-bench Verified, where cost per task drops from 20.0 to 9.7 cents and score rises from 40.0 to 43.2. The worst is MiniMax-M2.7 on BrowseComp-Plus, which scores 58.9 with AKV — 3.4 points below Text Engineering’s 62.3.

Serving speed and agent-directed policies

Serving tests ran GPT-OSS-120B on 80 long-context SWE-bench Verified tasks using 8 A800-80GB GPUs in BF16. At each method’s peak setting, the authors report AKV delivers 1.18× output throughput and 1.53× input throughput, with P95 request latency 1.51× faster. At concurrency 16, Standard’s P95 latency reaches 229.90 seconds versus 30.63 seconds for AKV because KV pressure pushes Standard into repeated re-prefill. Full-attention time per decode step falls from 7.082 ms to 2.849 ms.

The API is stateless and OpenAI-compatible: clients send the full message history plus optional drop_message and reposition fields. A CachePolicy builds those fields from a trigger, a selector, and a reposition rule, with built-in turn-based, token-based, or model-based policies; a CacheTool lets the model call drop_context itself. MiniMax-M2.7 refined policies on 20 BrowseComp-Plus tasks and then ran 100 held-out tasks. The iteration-6 policy cut cost 17.8% versus Standard at baseline accuracy, and the abstract reports 7.1% savings beyond the expert-designed policy. Exposing cache control as a tool lifted GPT-OSS-120B from 52.1 to 61.0 on a 100-task subset, although MiniMax-M2.7’s cost rose from 5.4 to 5.9 cents.

AI Mastery analysis

AKV shifts the optimization target from summarization quality to cache-state management. The implementation consequence is that it works only where the serving stack understands drop and reposition nodes — a modified SGLang deployment, not a portable prompt or harness change. Support is currently limited to Qwen, GPT-OSS, and MiniMax-M2 text architectures, with linear attention listed as future work. The system is self-hosted only, and every figure comes from the authors on one 8× A800-80GB BF16 setup. The high-concurrency AKV runs were measured through the first 30 task completions, while Standard used a one-hour window, so the latency comparison is not exactly like-for-like.

Policy choice matters more than the cache mechanism alone. MarkTechPost notes dropping reasoning cache on SWE-bench Verified fell to 36.5, dropping both to 34.5, versus 41.0 for tool drop, and MiniMax-M2.7’s BrowseComp-Plus score was below Text Engineering. That fits the pattern in Infrastructure Rewrites, Not Model Weights, Drive 2026 AI Gains: serving-layer control points deliver cost shifts only when the selection policy matches the workload.

If the 27.7% average cost reduction holds outside the author-run benchmark, AKV gives SGLang operators a concrete way to stretch GPU budgets on long agent traces without rewriting history. Long-running agent context is becoming a cache-placement problem rather than a summarization problem.

Sources

• Primary: GitHub - sgl-project/sglang: SGLang is a high-performance serving framework for large language models and multimodal models. · GitHub
• Reported by: MarkTechPost: Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang