Why does increasing max-num-seqs cause a runtime CUDA OOM?

General
1

I am running the Gemma model using vLLM and trying to understand how --gpu-memory-utilization and --max-num-seqs affect GPU memory.

My understanding is that GPU memory includes:

  • Model weights
  • KV cache
  • Activation memory
  • CUDA graph memory
  • Matrix multiplication and kernel workspaces

During startup, vLLM reports:
Model loading took 25.5 GiB memory

Estimated CUDA graph memory: 0.50 GiB total

Available KV cache memory: 12.17 GiB

CUDA graph pool memory: 0.45 GiB

However, when I increase --max-num-seqs, the server eventually encounters a CUDA OOM:

max-num-seqs 10 → Successful

max-num-seqs 20 → Successful

max-num-seqs 50 → Successful

max-num-seqs 70 → Successful

max-num-seqs 90 → Successful

max-num-seqs 92 → CUDA OOM

max-num-seqs 100 → CUDA OOM

  1. Why does runtime GPU memory consumption increase with --max-num-seqs if the model weights and KV-cache allocation are already determined during startup?
  2. Is the additional memory primarily caused by activations, temporary kernel workspaces, CUDA graph capture sizes?
  3. Is there a recommended method or formula for selecting a safe combination of:
    • --gpu-memory-utilization
    • --max-num-seqs
    • --max-num-batched-tokens
3

@RunLLM can you answer please