Is it true that In MoonCake, the prefix information disappears when it reaches the storage? Can the prefix information be preserved for use by the eviction algorithm?
No, it is not true that prefix information necessarily disappears in Mooncake storage. The MooncakeStoreConnector in vLLM preserves prefix information by using hash-based deduplication: each KV cache block is identified by a hash of its prefix tokens and other metadata, and this hash is used as the storage key in Mooncake. This allows the prefix information to be reconstructed or referenced for cache management, including eviction policies, as long as the hash and associated metadata are maintained in the key namespace and connector logic. However, as of now, Mooncake’s internal eviction policy is handled on the store side, and vLLM does not directly control or use prefix information for eviction within Mooncake itself, though the connector design allows for such extensions in the future. See MooncakeStoreConnector Usage Guide and MooncakeStoreECConnector PR.
Would you like more detail on how the prefix hash and metadata are constructed or used for eviction?
Sources:
Hint: Mention @Herald in the post for followups.
@Herald yes, please provide more detail
Certainly! In vLLM, when using MooncakeStoreConnector, each KV cache block is assigned a unique hash key. This hash is computed from the sequence of prefix tokens, parent block hash, and extra metadata (such as LoRA IDs, multimodal hashes, and cache salt for isolation). This hash key is then used as the storage key in Mooncake, meaning the prefix information is encoded into the key itself and does not disappear when stored externally. The key structure (see PoolKey and KeyMetadata) includes model name, parallel ranks, group ID, and the chunk (prefix) hash, ensuring that the prefix context is always recoverable from the key string (MooncakeStoreConnector Usage Guide, API reference).
For eviction, Mooncake itself manages its own internal policies (e.g., LRU, lease-based, or group-aware eviction), and these policies operate on the stored keys. While vLLM does not directly instruct Mooncake to evict based on prefix semantics, the grouping and key structure (including group semantics and cache prefix) allow Mooncake to apply group-aware eviction if configured (PR: Add store group semantics). Thus, prefix information is preserved and can be leveraged for advanced eviction strategies, but the actual use of prefix-aware eviction depends on Mooncake’s configuration and connector capabilities.
Sources:
Hint: Mention @Herald in the post for followups.