# On prefill-only, kv\_cache\_usage\_perc reaches max 0.32?

**URL:** <https://discuss.vllm.ai/t/on-prefill-only-kv-cache-usage-perc-reaches-max-0-32/2314>\
**Category:** Benchmarking\
**Created:** [February 3, 2026, 2:04pm UTC](https://discuss.vllm.ai/t/on-prefill-only-kv-cache-usage-perc-reaches-max-0-32/2314 "2026-02-03T14:04:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![biranofer](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/biranofer/32/665_2.png) [@biranofer](https://discuss.vllm.ai/u/biranofer)\
**Post date:** [February 3, 2026, 2:04pm UTC](https://discuss.vllm.ai/t/on-prefill-only-kv-cache-usage-perc-reaches-max-0-32/2314/1 "2026-02-03T14:04:59Z")

</div>

Would appreciate referring to the phenomena described in

> <https://github.com/llm-d-incubation/workload-variant-autoscaler/issues/662>
>
> The KVCache percentage metric presents at some scenarios unexpected behavior. E.…g., on prefill-only experiment:  
> \- GPU H100
> \- Input tokens per request: 6000
> \- Output tokens: 2
> \- Requests per second: 8
> \- Max-Tokens-in-Batch: 65000
> 
> \<img width="1268" height="1023" alt="Image" src="https://github.com/user-attachments/assets/f31f2649-c9de-444d-a590-4616045d385e" /\>
> 
> We see in the above experiment graph that we are clearly in saturation state (num\_requests\_waiting is constantly growing), the num\_requests\_running stabilized on ~20, and the kv\_cache\_usage\_perc stabilized on just ~0.32 (32%), so in this scenario the KVCache will not trigger a saturation indication. An interesting question is how the num\_requests\_running got to 20 - with 6000 tokens per request it sums up to 120000 - above the 65000 (Max-Tokens-in-Batch) - this is in vLLLM 14.0 (and different than was in e.g. vLLM 11.0).
> 
> When running the same experiment, just with output tokens per request 200 (i.e. combining prefill and decode tasks), we get this result:
> 
> \<img width="1275" height="1046" alt="Image" src="https://github.com/user-attachments/assets/6f0e1443-9560-46b3-8835-a4364e993743" /\>
> 
> Here the num\_requests\_running got to ~60 - this is since decode requests are counted as just 1 token, so many more requests can be scheduled. This time the kv\_cache\_usage\_perc gets to almost 1 (100%).
> 
> Better understanding of these metrics can be used to refine the saturation-based algorithm, providing better prediction and proactive scaling operations.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [February 3, 2026, 2:05pm UTC](https://discuss.vllm.ai/t/on-prefill-only-kv-cache-usage-perc-reaches-max-0-32/2314/2 "2026-02-03T14:05:38Z")

</div>

The phenomena described in [Better understanding of the kv\_cache\_usage\_perc and num\_requests\_running metrics can be used to refine the saturation-based algorithm · Issue #662 · llm-d-incubation/workload-variant-autoscaler · GitHub](https://github.com/llm-d-incubation/workload-variant-autoscaler/issues/662) involve unexpected behavior of the kv\_cache\_usage\_perc and num\_requests\_running metrics in vLLM under certain load conditions. Specifically, even when the system is saturated (num\_requests\_waiting grows and num\_requests\_running stabilizes), the kv\_cache\_usage\_perc may remain low (e.g., ~32%), indicating that cache usage alone may not reliably signal server saturation. This suggests that autoscaling algorithms should consider both metrics together, not just kv\_cache usage, to accurately detect and respond to saturation states.

Would you like more detail on how these metrics interact or how to use them for autoscaling?

Sources:

- [Better understanding of the kv\_cache\_usage\_perc and num\_requests\_running metrics can be used to refine the saturation-based algorithm · Issue #662 · llm-d-incubation/workload-variant-autoscaler](https://github.com/llm-d-incubation/workload-variant-autoscaler/issues/662)

Hint: Mention @RunLLM in the post for followups.