Should request priority also influence KV cache retention?

General
1

Hi vLLM team,

Thanks for the great work on vLLM.

I have a design question regarding the relationship between request priority scheduling and KV cache management.

As I understand it, vLLM currently supports request priorities for scheduling, where higher-priority requests are selected earlier from the waiting queue.

However, once a request starts running and its KV cache blocks are allocated, it seems that the KV cache manager no longer distinguishes between high- and low-priority requests. During cache eviction, blocks are selected according to the cache management policy, without considering the priority of the request that owns them.

This made me wonder whether this separation is an intentional design choice.

For example, suppose we have two requests:

  • Request A (priority = high)
  • Request B (priority = low)

Both have generated KV cache on the GPU. If GPU memory becomes full, KV blocks belonging to Request A could potentially be evicted before those belonging to Request B, resulting in recomputation and additional latency for the high-priority request.

From a QoS perspective, one could argue that request priority should perhaps influence not only scheduling, but also KV cache retention.

On the other hand, I can also imagine that introducing request priority into cache eviction may reduce overall cache reuse or hurt system throughput, so there are likely important trade-offs involved.

I also came across RFC #37003 on context-aware KV cache retention. My understanding is that the proposed retention priority is primarily motivated by expected future reuse, which seems different from request-level scheduling priority.

I’m curious about the maintainers’ thoughts on the following questions:

  1. Was it an intentional design decision to keep request scheduling priority and KV cache management independent?
  2. Have you considered allowing eviction policies to optionally incorporate request priority?
  3. Do you see request priority and retention priority as two orthogonal concepts, or could they eventually be combined within a unified cache management policy?

I’d really appreciate any insights into the design rationale. Thanks!

2

You are correct: in current vLLM releases, request priority only affects scheduling order (which requests are admitted to run), not KV cache eviction. Under cache pressure, high-priority requests are not protected from losing their cached KV blocks—eviction is based on LRU or other cache policies, not request priority. This separation is intentional and documented as a known limitation, though there is active discussion and ongoing work to make cache eviction optionally priority-aware in the future. For example, this issue and this PR show that priority-aware KV cache eviction is being developed, so that high-priority requests can retain their cache blocks longer under pressure.

Request priority (for scheduling) and retention priority (for cache eviction) are currently orthogonal in vLLM, but the maintainers recognize that combining them could improve QoS for high-priority workloads. The Context-Aware KV-Cache Retention API RFC proposes a flexible mechanism for annotating blocks with eviction priorities, which could be set based on request priority or other signals. This would allow future policies to unify or coordinate scheduling and cache retention priorities if desired. Would you like a step-by-step breakdown of how these mechanisms work and interact?

Sources:

Hint: Mention @Herald in the post for followups.