STAR-KV replaces a transformer’s full-dimension key/value cache with a per-layer low-rank projection (head-wise decomposition for K, joint decomposition for V), calibrated offline via short knowledge distillation, and stored in a compressed paged format. This repo reproduces that pipeline and wires it into vLLM’s V1 continuous-batching, paged-attention serving stack via dedicated Triton kernels.
Your summary is accurate: Proxima’s STAR-KV implements low-rank KV cache compression (head-wise for keys, joint for values), calibrated offline, and integrates with vLLM’s V1 paged-attention stack using custom Triton kernels. This approach allows vLLM to serve more concurrent requests and longer contexts within the same GPU memory, as demonstrated by 4x concurrency improvements and 3x+ memory efficiency in real hardware tests (Proxima GitHub).
Would you like more detail on STAR-KV’s integration with vLLM, the calibration process, or the measured results?
