Decode KV writeback: can multi-turn chat preserve cache reuse with default templates?

Features KV-Cache
1

Hi vLLM community,

I would like to understand the practical benefits of Decode-phase KV writeback when server-side chat templates change the token sequence between conversation turns.

This relates to the Decode-Phase KV Cache Put item in the Mooncake Store Connector roadmap #45036 and the related work in #52466, #53129, and #54307.

  • #52466, merged: adds Decode KV offloading, including consumer-side writeback with save_decode_cache.

  • #53129, merged: enables heterogeneous-TP Store sharing, including Decode writeback into a common Store layout.

  • #54307, currently open: extends heterogeneous-TP sharing to hybrid KV caches.

Assume the client faithfully replays all conversation history, including reasoning where applicable. The concern is what happens after vLLM receives those messages and applies the chat template.

I tested the process with Qwen3. The result is as follower:

Non-thinking mode: Qwen3

The affected default template adds <think>\n\n</think>\n\n before generating an answer, but omits it when rendering that answer as historical assistant content in the next request:

Turn 1 execution:
System | User1 | <think>\n\n</think>\n\n | Answer1

Turn 2 input after template rendering:
System | User1 | Answer1 | User2 | <think>\n\n</think>\n\n

qwen3_nonthinking_four_turn_kv_reuse
qwen3_nonthinking_four_turn_kv_reuse1680×1120 105 KB

This behavior is also reported in Qwen3 issue #1826.

Thinking mode: historical reasoning removed

For Qwen3 and DeepSeek V4 configurations that discard historical reasoning, the template can remove the reasoning even when the client supplies it:

Turn 1 execution:
System | User1 | <think>Reasoning1</think> | Answer1

Turn 2 input after template rendering:
System | User1 | Answer1 | User2 | ...

DeepSeek V4 provides drop_thinking, with retention options and tool-use exceptions; this example concerns the path where historical reasoning is dropped. See the official encoding reference.

thinking_four_turn_kv_reuse
thinking_four_turn_kv_reuse1680×1180 124 KB

In both cases, Decode may successfully save KV, but the next Prefill receives a different token prefix. Complete blocks before the first difference can still hit; the affected block and subsequent prefix-dependent blocks cannot reuse that Decode execution’s KV. Identical final-answer text is insufficient because its KV was computed with a different preceding context.

Questions

  1. Does vLLM provide, or plan to provide, a server-side path that preserves the previous Decode token prefix when rendering subsequent chat turns? With complete history replay, can enabling Decode KV writeback lead to next-turn reuse under default templates, or are template/configuration changes a prerequisite?

  2. What next-turn reuse and performance benefits have been demonstrated with Decode writeback enabled for these models? In particular, do the demonstrated results in PR include normal multi-turn Chat API requests processed by the default templates, with cache hits extending into the previous answer’s generated tokens?

  3. When historical reasoning is intentionally removed server-side, what useful scenarios remain for Decode KV writeback? If ordinary next-turn reuse is limited, which supported workloads are expected to benefit from saving those generated KV blocks?

I would appreciate clarification on the intended server-side behavior and the conditions under which Decode KV writeback delivers practical benefits.

2

Decode-phase KV writeback only enables next-turn reuse if the exact token sequence—including all template-injected tokens like <think> or reasoning—is preserved between turns. With default Qwen3 and DeepSeek V4 templates, server-side chat rendering often strips or alters these tokens (e.g., omitting <think> or dropping reasoning), so the next turn’s prompt diverges from the previous decode’s output. As a result, only the prefix up to the first difference can hit the cache; answer tokens computed after a changed prefix (like a missing <think>) cannot be reused, and Decode KV writeback provides little or no benefit in standard multi-turn chat flows using these templates. This is confirmed in Qwen3 issue #1826 and the DeepSeek encoding reference.

vLLM does not currently provide a server-side mechanism to force preservation of the previous decode’s token prefix in subsequent turns under default templates. Practical reuse of Decode-phase KV blocks across turns requires either (a) a template/configuration that faithfully replays all generated tokens (including reasoning/thinking markers), or (b) a custom workflow where the client reconstructs the exact prior output as input. Demonstrated performance benefits in PRs like #52466 and #53129 are limited to scenarios where such exact replay is possible; for standard multi-turn Chat API usage with default templates, cache hits do not extend into previous answers’ generated tokens. When historical reasoning is intentionally removed, Decode KV writeback is only useful for specialized workloads (e.g., tool-calling, agentic flows, or custom templates that preserve all tokens). For most default chat scenarios, its practical benefit is minimal unless template or client behavior is changed.

Would you like a step-by-step breakdown of the cache hit logic or example code for custom template handling?

Sources:

Hint: Mention @Herald in the post for followups.