AI News
All stories
Reported

Agent KV cache study reports 27.7% lower costs without history summarization

A proposed AKV system drops and repositions cache entries directly instead of summarizing history, and reports 27.7% lower costs and 1.7 points higher benchmark scores.

Research · · 2 min read

A paper posted to arXiv on October 20, 2026 proposes letting an LLM agent manage its own key-value cache directly, rather than shrinking its conversation history through summarization. The authors, Tianyu Fu and eight co-authors, report that one policy built on this approach lowers average cost by 27.7% and raises average benchmark scores by 1.7 points compared with conventional threshold-triggered summarization.

The problem comes from how LLM serving works. When an agent runs a long task, its observations and actions accumulate, and the KV cache that stores attention state for each token grows with them. Trajectories can also exceed the model's maximum context length. The prior approach, which the abstract calls context engineering, is to trim or summarize the history. The catch is that the cache is built on a prefix. Rewriting an early part of the history changes every cached state after it. The server must then re-prefill the retained context, paying for that computation again. Summarizing too aggressively to avoid that cost can also hurt accuracy, since task accuracy suffers when too little context is kept.

The paper's alternative, which it calls cache engineering, leaves the text alone and changes only the cache. Two operations do the work. Drop excludes selected cache entries from future attention. Reposition adjusts the position encodings of the retained keys so the remaining entries close the gaps that drops leave behind. The authors also point out a serving problem this creates. Identical token sequences can produce different cache states depending on which cache operations ran before them, so ordinary prefix matching is not enough to decide whether a cache can be reused. Their AKV serving engine handles this internally while exposing a stateless interface that works with existing serving APIs.

The public repository for the project, thu-nics/AKV, shows how this looks in practice. The example there keeps tool results from the latest completed turn. After processing a new message, it sends a drop_message field naming an earlier tool result to remove, and a reposition field to compact positions. The repository's README notes that the server applies these operations and that the client should keep the full history with its original numbering. The repository says drop and reposition are currently enabled for specific text-model architectures, including Qwen, Qwen3 MoE variants, GPT-OSS and MiniMax-M2.

The 27.7% reduction is the average for a tool-response rolling-drop policy compared with threshold-triggered summarization. That is a specific baseline, not every alternative. The abstract also reports up to 1.2x output throughput and a 1.5x speedup in P95 request latency with AKV. A separate result says recursive self-improvement of cache policies cuts cost a further 7.1% over expert-designed policies.

What changed

A proposed AKV approach lets an LLM agent drop and reposition its KV cache entries directly instead of summarizing conversation history, reporting 27.7% lower average cost than threshold-triggered summarization.

Type: Research · Applies to: AKV

Why it matters

Long-running agent sessions are costly to serve, so cache-level control could cut serving bills if the reported gains hold outside the study.

Original source

Where we saw it

1 source

Demetrios from the AAIF

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works