Running out of memory despite very low `--gpu-memory-utilization`

General
1

I am trying to run DiffusionGemma. I only have a 16GB AMD GPU so I expect to be doing CPU offloading. I used the following command:

vllm serve --port 8085 google/diffusiongemma-26B-A4B-it --max-num-seqs 1 --enable-auto-tool-choice --tool-call-parser gemma4 --chat-template ~/Downloads/tool_chat_template_gemma4.jinja --reasoning-parser gemma4 --cpu-offload-gb 100 --gpu-memory-utilization 0.1

This fails with:

(EngineCore pid=298899)

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 968.00 MiB. GPU 0 has a total capacity of 15.98 GiB of which 354.00 MiB is free. Of the allocated memory 15.09 GiB is allocated by PyTorch, and 293.03 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management ( Redirecting… )

I don’t understand the issue:

  1. --cpu-offload-gb 100 should be enough for the full model on its own.
  2. --gpu-memory-utilization 0.1 is being blown past and I don’t know why.

I am running:

% vllm -v
0.24.0+rocm725
2

The error occurs because, even with --cpu-offload-gb set, vLLM still requires some GPU memory for model weights, activations, and especially the KV cache, and the offloading mechanism may not fully offload all components for very large models. Also, --gpu-memory-utilization only limits the portion of GPU memory used for the KV cache, not for all allocations, so total usage can exceed this value during model initialization or CUDA graph capture. This is a known limitation, especially on GPUs with limited VRAM and when running very large models like DiffusionGemma-26B-A4B-it, even with large CPU offload settings. See the explanations in docs.vllm.ai CacheConfig and vllm-project/vllm/issues/23049.

Additionally, some users report that vLLM’s offloading is not always honored as expected, and OOM can still occur if the model or context length is too large for the available GPU memory, regardless of offload settings (vllm-project/vllm/issues/9367, vllm-project/vllm/issues/32653). You may need to further reduce --max-model-len, --max-num-seqs, or try a smaller model. Would you like a step-by-step breakdown of how vLLM allocates memory and why offloading may not be sufficient in this case?

Sources:

Hint: Mention @Herald in the post for followups.

3

This just seems to be a bug related to DiffusionGemma. Similarly large models with the same settings work fine.