Is there any roadmap to support prefix caching on dram and disk?

[hmellor]

View in #questions on Slack

@Zeyu: Hi guys! I wonder is there any roadmap to support prefix caching on dram and disk?

@Harry_Mellor: roadmap.vllm.ai

@Yuhan_Liu: hey @Zeyu feel free to also check out the Production Stack project (#C089SMEAKRA) which already supports prefix caching on dram and disk . (Production Stack is supported by vLLM as a first party project) Here is an example of offloading KV to CPU memory: https://github.com/vllm-project/production-stack/blob/main/tutorials/05-offload-kv-cache.md

@Zeyu: Thanks guys!

[RunLLM]

No, that is not correct. LMCache supports both local CPU (DRAM) and local disk offloading together. You can configure both LMCACHE_LOCAL_CPU and LMCACHE_LOCAL_DISK with their respective size limits, and LMCache will use DRAM first and then spill over to disk as needed. Example usage and environment variable setup for both are shown in the LMCache example code and discussed in user reports (see here).

Would you like more detail on how to configure both DRAM and disk offloading in LMCache?

Sources:

• Issue: failed to run LMCache example for v0
• LMCache example code