On looking into the code, I figure that both the implementations have different backends. vLLM fp8 KV Cache Quantization is in the Flash Attention Backend, which has highly optimized CUDA Kernels for CUDA GPUs. TurboQuant is implemented in a separate TurboQuant Backend, which has Triton kernels.
So the comparison does not seem to be fair. Am I missing something here?
Moreover, are there planned efforts to implement TQ in Flash Attention backend.
You are correct: in vLLM, FP8 KV cache quantization is implemented in the Flash Attention backend (using highly optimized CUDA kernels), while TurboQuant (TQ) is implemented in a separate TurboQuant backend using Triton kernels. This means the performance and throughput comparisons in the referenced blog post are not strictly apples-to-apples, as each backend has different kernel optimizations and architectural tradeoffs. The blog itself notes that FP8 is the best default for most workloads, partly due to its mature, highly optimized backend, while TurboQuant offers higher compression but with a throughput penalty due to its more complex dequantization path and less mature kernel optimizations (vLLM TurboQuant blog).
As of now, there is no explicit mention in the official documentation or codebase of planned efforts to implement TurboQuant directly within the Flash Attention backend for CUDA GPUs. TurboQuant remains a standalone backend, and the current design isolates it from the Flash Attention path (TurboQuantAttentionBackend source). If you want more detail on backend selection or future roadmap, let me know.