FlashMLA issue when running FP8 Deepseek V8 model on H20

General
1

When running deepseek v3 fp8 model on H20 using vllm 0.9.2, the following error is reported, does anyone know if fp8 kv cache is supported? Appreciate if anyone can elaborate why it is not supported, and what are the options? like:

  • Disable fp8 kv cache
  • Disable flashmla
lightning_vllm-1  | (VllmWorker rank=4 pid=124) INFO 09-08 19:30:24 [cuda.py:228] Using FlashMLA backend on V1 engine.lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487] WorkerProc failed to start.lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487] Traceback (most recent call last):lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py”, line 461, in worker_mainlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     worker = WorkerProc(*args, **kwargs)lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py”, line 358, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.worker.load_model()lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py”, line 185, in load_modellightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.model_runner.load_model()lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py”, line 1776, in load_modellightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.model = model_loader.load_model(lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                  ^^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py”, line 38, in load_modellightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     model = initialize_model(vllm_config=vllm_config,lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py”, line 64, in initialize_modellightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     return model_class(vllm_config=vllm_config, prefix=prefix)lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py”, line 723, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.model = DeepseekV2Model(vllm_config=vllm_config,lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py”, line 152, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py”, line 661, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.start_layer, self.end_layer, self.layers = make_layers(lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                                                     ^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py”, line 640, in make_layerslightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py”, line 663, in lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     lambda prefix: DeepseekV2DecoderLayer(lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                    ^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py”, line 549, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.self_attn = attn_cls(lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                      ^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v2.py”, line 471, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.mla_attn = Attention(lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                     ^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py”, line 140, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     self.impl = impl_cls(num_heads, head_size, scale, num_kv_heads,lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]   File “/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/backends/mla/flashmla.py”, line 151, in initlightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487]     raise NotImplementedError(lightning_vllm-1  | (VllmWorker rank=3 pid=123) ERROR 09-08 19:30:25 [multiproc_executor.py:487] NotImplementedError: FlashMLA V1 with FP8 KV cache not yet supportedlightning_vllm-1  | [rank0]:[W908 19:30:25.194794771 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see  (function operator())
2

FP8 KV cache is not supported with FlashMLA backend in vLLM V1 engine, including for DeepSeek V3 on H20. The error NotImplementedError: FlashMLA V1 with FP8 KV cache not yet supported is raised because the implementation is missing for this combination, as confirmed in the FlashMLA backend source and issue reports.

To work around this, you can:

  • Disable FP8 KV cache by not setting --kv-cache-dtype fp8 (use default or supported dtype).
  • Switch backend: If you require FP8 KV cache, use a backend that supports it (e.g., FlashInfer for some models/hardware), but FlashMLA does not support it yet.
  • There is no current support for FP8 KV cache with MLA models (like DeepSeek V3) in vLLM V1; support is planned but not available yet.

Would you like more detail on how to change your configuration or backend?

Sources:

3

FP8 KV cache support for the FlashMLA backend was added recently ( [Kernel] Add FP8 support with FlashMLA backend by MatthewBonanni · Pull Request #22668 · vllm-project/vllm · GitHub ) and will be included in the next release. If you’d like to use it now you can try installing the latest code from main!

4

Correct—FP8 KV cache support for the FlashMLA backend was recently merged and will be available in the next vLLM release. To use it immediately, install vLLM from the latest main branch (see PR #22668). This enables FP8 KV cache with FlashMLA, improving throughput on supported hardware.

If you need this feature now, clone the vLLM repository and install from source. Would you like step-by-step instructions for installing from main?

Sources: