I’m sure this has been asked, but what is the proper way to use quant k,v cache on ampere (3090’s)? –kv-cache-dtype fp8_e4m3 (or fp8) does not seem to save any VRAM and the seq length doesn’t increase. Thanks!

On NVIDIA Ampere GPUs (like 3090), vLLM’s FP8 KV cache quantization (--kv-cache-dtype fp8 or fp8_e4m3) is not supported due to hardware and Triton kernel limitations. Attempting to use these options will not reduce VRAM usage or increase sequence length, and may result in errors or fallback to unqua…

You’re welcome! If you have any more questions about vLLM or quantization in the future, feel free to ask. Sources: Quantized KV Cache — vLLM (stable) vLLM GitHub Issue: Unable to use fp8 kv cache with chunked prefill on ampere vLLM GitHub PR: Add FP8 K/V Scale and dtype conversion for prefix/pre…

KV Cache quantizing?

Hardware Support NVIDIA GPU Support

Frenzy_Biscuit June 2, 2025, 2:33pm 3

No, thank you though!

Topic		Replies	Views
NVFP4 Support In Attention KV-Cache	1	842	March 16, 2026
Questions about KV Cache Quantization General	1	1043	June 30, 2025
RDNA4 FP8 support General	1	219	May 2, 2026
Flash Attention 3 FP8 Support General	1	795	April 30, 2025
Native FP8 WMMA Support for AMD RDNA4 (RX 9070 XT / R9700) in vLLM General	5	2543	February 12, 2026

KV Cache quantizing?

Related topics