# Which arguments affect GPU memory

**URL:** <https://discuss.vllm.ai/t/which-arguments-affect-gpu-memory/551>\
**Category:** General\
**Created:** [April 29, 2025, 4:51am UTC](https://discuss.vllm.ai/t/which-arguments-affect-gpu-memory/551 "2025-04-29T04:51:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shawn](https://avatars.discourse-cdn.com/v4/letter/s/f6c823/32.png) [@Shawn](https://discuss.vllm.ai/u/Shawn)\
**Post date:** [April 29, 2025, 4:51am UTC](https://discuss.vllm.ai/t/which-arguments-affect-gpu-memory/551/1 "2025-04-29T04:51:56Z")

</div>

Why does each GPU use more than 15GB of memory when running the DeepSeek 14B INT4 model on 4 x 16GB Intel ARC 770 GPUs with tensor parallelism (TP) set to 4 ?

DeepSeek 14B INT4 model weight is only 7-8G around.

Can we know which arguments will affect the GPU memory?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [April 29, 2025, 4:52am UTC](https://discuss.vllm.ai/t/which-arguments-affect-gpu-memory/551/2 "2025-04-29T04:52:19Z")

</div>

The memory usage on each GPU can be influenced by several factors beyond just the model weights. According to [vLLM documentation](https://docs.vllm.ai/en/stable/serving/engine_args.html#named-arguments), the `gpu_memory_utilization` parameter controls the fraction of GPU memory used for the model executor, which includes model weights, activations, and KV cache. Additionally, enabling CUDA graphs or using certain quantization methods can increase memory usage.

The `tensor_parallel_size` argument affects how the model is distributed across GPUs, and the `max-model-len` and `max-num-seqs` parameters can also impact memory usage by determining the size of the KV cache. If the `gpu_memory_utilization` is set too high, it may lead to out-of-memory errors due to insufficient space for activations and other runtime memory needs. Adjusting these parameters can help manage memory usage more effectively.

Would you like more detail on how to adjust these parameters for optimal memory usage?

Sources:

- [vLLM documentation](https://docs.vllm.ai/en/stable/serving/engine_args.html#named-arguments)

---

<div class="post-metadata">

**Author:** ![Ranjith](https://avatars.discourse-cdn.com/v4/letter/r/8797f3/32.png) [@Ranjith](https://discuss.vllm.ai/u/Ranjith)\
**Post date:** [September 3, 2025, 6:14am UTC](https://discuss.vllm.ai/t/which-arguments-affect-gpu-memory/551/3 "2025-09-03T06:14:23Z")

</div>

@RunLLM, Yes can you suggest some

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [September 3, 2025, 6:14am UTC](https://discuss.vllm.ai/t/which-arguments-affect-gpu-memory/551/4 "2025-09-03T06:14:54Z")

</div>

Certainly! To reduce GPU memory usage in vLLM, you can:

1. **Lower `--gpu-memory-utilization`** : This parameter (default 0.9) controls the fraction of GPU memory vLLM will use. Lowering it (e.g., to 0.7) reserves less memory for vLLM, but if set too low, you may get errors about insufficient memory for cache blocks.
2. **Reduce `--max-model-len`** : This limits the context length, directly reducing the size of the KV cache.
3. **Reduce `--max-num-seqs`** : This limits the maximum batch size, further reducing memory usage.
4. **Use `--enforce-eager`** : Disables CUDA graph compilation, which can save memory at the cost of some performance.
5. **Use quantized models** : INT4 models already help, but ensure you are not using additional features that increase memory.
6. **Adjust `max_num_batched_tokens`** : Lowering this can reduce peak memory usage during batch processing.

Example:

```python
from vllm import LLM
llm = LLM(
    model="deepseek-ai/DeepSeek-14B-INT4",
    tensor_parallel_size=4,
    gpu_memory_utilization=0.7,
    max_model_len=2048,
    max_num_seqs=2,
    enforce_eager=True
)

```

For more details on these arguments, see [Reducing memory usage](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size) and [CLI arguments](https://docs.vllm.ai/en/latest/cli/index.html#-gpu-memory-utilization).

Would you like a deeper explanation of how each parameter affects memory?

Sources:

- [Reducing memory usage](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size)