I’m trying to build a side project in the inference space. I’ve been talking to a few inference engineers and startups and I’ve been hearing how annoying it is to not have manual control over the KV cache at times and just constantly being subject to the black box caching methods of their inference providers. It is particularly annoying for agent swarms when you want to fork agents from the same cached prefix or manually store a cache for a longer period for a future agent to hit later.
Im curious if this problem is consistent across multiple people and if there are any solutions for it that people know about.
Manual control over the KV cache in vLLM is a frequently requested feature, especially for advanced use cases like agent swarms, but is not natively supported. vLLM’s current caching is automatic and LRU-based, with no official API for manually pinning, expiring, or injecting KV cache entries, though there are proposals and discussions about adding such features (e.g., pinned caching, explicit APIs for cache management) but these are not yet merged or available in stable releases. See discussions in vllm-project/vllm#8333 and vllm-project/vllm#10515.
Some workarounds exist, such as using prefix caching with cache salting for isolation, or leveraging external KV cache stores like LMCache or Mooncake for sharing across instances, but these still lack fine-grained manual control at the user level (LMCache example, Mooncake RFC). Would you like more detail on current workarounds or the status of proposed manual control features?
If you self-host, two of the three things you describe are already there. The third is mostly a memory-budget question, and some numbers help size it:
Forking agents from one prefix: with prefix caching (on by default, enable_prefix_caching), requests that start with the same tokens share the same KV blocks inside one vLLM instance. Ten agents forked from a 32K system prompt + tool history store that prefix once, not ten times.
Keeping a cache for later: KV offloading to CPU RAM (--kv-offloading-size <GiB>, backend native or lmcache via --kv-offloading-backend) keeps evicted prefix blocks in host memory, so an agent that comes back later can reuse them instead of re-running prefill.
How much you can keep: one 32K-token prefix at bf16 KV is 4 GiB for Llama-3.1-8B (128 KiB/token) and 10 GiB for Llama-3.1-70B (320 KiB/token). So a 64 GiB offload buffer holds about 16 such 8B prefixes or 6 for the 70B, and fp8 KV doubles both. Models with GQA-heavy or hybrid attention cache far less per token, which matters more than anything else for swarms.
What’s genuinely missing is explicit control: pinning a prefix so it isn’t evicted, or an API to say “keep this for 24 h”. Today eviction is automatic, so the practical lever is sizing the GPU pool plus offload buffer for your working set.