# Deploy a big LLM when GPU VRAM not enough **URL:** \ **Category:** General\ **Created:** [August 11, 2025, 12:08pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354 "2025-08-11T12:08:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 11, 2025, 12:08pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/1 "2025-08-11T12:08:29Z")
In Kubernetes environment, we want to deploy a very large LLM model that uses over 200B parameters. It may need over 500GB VRAM. In our node we have available only 192GB VRAM, 4 GPUs in the same node, can we use CPU offloading and utilize the node’s RAM to deploy the model? Sharing between RAM and VRAM. What parameters can be passed in the Kubernetes manifest to achieve this? Of course, inference performance will be very low ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 11, 2025, 12:08pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/2 "2025-08-11T12:08:52Z")
Yes, you can use CPU offloading in vLLM to deploy models larger than your available GPU VRAM by utilizing system RAM. The key parameter is `--cpu-offload-gb`, which specifies how many GB of CPU RAM to use for offloading model weights. For example, with 192GB VRAM and 4 GPUs, you can set `--tensor-parallel-size 4` and add `--cpu-offload-gb` to increase the effective memory available for the model. Note that this requires a fast CPU-GPU interconnect and will significantly reduce inference throughput and increase latency, as weights are streamed between CPU and GPU during inference. Example manifest args: ```yaml args: - "--model" - "your-model-name" - "--tensor-parallel-size" - "4" - "--cpu-offload-gb" - "400" ``` You should adjust `--cpu-offload-gb` to match the amount of RAM you want to allocate for offloading. For more details, see the [official documentation](https://docs.vllm.ai/en/latest/getting_started/examples/basic.html) and [related discussions](https://github.com/vllm-project/vllm/issues/3563). Would you like a full Kubernetes YAML example or more details on performance implications? Sources: - [vLLM Basic Example: CPU offload](https://docs.vllm.ai/en/latest/getting_started/examples/basic.html) - [vLLM GitHub Issue: Offload Model Weights to CPU](https://github.com/vllm-project/vllm/issues/3563) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 11, 2025, 12:37pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/3 "2025-08-11T12:37:13Z")
Can I use AWQ instead? In that case, in Kubernetes manifest, do I need to pass any arguement? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 11, 2025, 12:37pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/4 "2025-08-11T12:37:35Z")
Yes, you can use AWQ quantized models with vLLM to reduce memory usage. In your Kubernetes manifest, you need to add the argument `--quantization awq` when launching vLLM with an AWQ model. For example: ```yaml args: - "--model" - "your-awq-model" - "--quantization" - "awq" ``` This tells vLLM to load and run the model in AWQ quantized mode. No other special arguments are required for AWQ, but you can combine this with `--cpu-offload-gb` if you still need to offload to CPU RAM. See [AutoAWQ documentation](https://docs.vllm.ai/en/latest/features/quantization/auto_awq.html) for more details. Would you like more detail or a full manifest example? Sources: - [AutoAWQ — vLLM](https://docs.vllm.ai/en/latest/features/quantization/auto_awq.html) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 11, 2025, 5:13pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/5 "2025-08-11T17:13:48Z")
For vLLM pods, using enough VRAM for FP16 or AWQ, what is the recommendation for RAM and CPU in pod resources? IS there any relation to the number of parameters to the VRAM used in total (e.g. 196GB VRAM)? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 11, 2025, 5:14pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/6 "2025-08-11T17:14:22Z")
It seems likely that for vLLM pods running in FP16 or AWQ mode (without CPU offloading), the main resource requirement is GPU VRAM, which should be large enough to fit the model weights and KV cache. The RAM and CPU requests in the pod spec are typically much lower—often 16–32GB RAM and 4–8 CPU cores are sufficient for serving, unless you have very high concurrency or multi-modal workloads. There is no strict formula, but RAM/CPU are not directly proportional to model parameter count; VRAM is the main constraint for model size, as model weights and cache must fit there for FP16/AWQ inference. For a 196GB VRAM node, you can generally run a model whose weights and cache fit within that VRAM, regardless of system RAM. Would you like more detail or a sample resource block for your manifest? Sources: - [https://docs.vllm.ai/en/latest/api/vllm/entrypoints/llm.html#vllm.entrypoints.llm.LLM](https://docs.vllm.ai/en/latest/api/vllm/entrypoints/llm.html#vllm.entrypoints.llm.LLM) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 13, 2025, 9:02am UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/7 "2025-08-13T09:02:01Z")
When the GPU VRAM is not enough, could we use AWQ quantization with cpu offload deploying the vllm in Kubernetes? Do we add the max RAM GB in arguements? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 13, 2025, 9:02am UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/8 "2025-08-13T09:02:33Z")
Yes, you can use AWQ quantization with CPU offload in vLLM to deploy large models when GPU VRAM is insufficient. In your Kubernetes manifest, add both `--quantization awq` and `--cpu-offload-gb ` arguments to specify AWQ mode and the amount of system RAM to use for offloading model weights. For example: ```yaml args: - "--model" - "your-awq-model" - "--quantization" - "awq" - "--cpu-offload-gb" - "256" ``` Set `` to the maximum RAM (in GB) you want vLLM to use for offloading. This enables vLLM to split model weights between VRAM and RAM, but performance will be lower due to CPU-GPU transfers. See [AutoAWQ documentation](https://docs.vllm.ai/en/latest/features/quantization/auto_awq.html) and [CPU offload usage](https://docs.vllm.ai/en/latest/getting_started/examples/basic.html). Would you like more detail or a full manifest example? Sources: - [vLLM Basic Example: CPU offload](https://docs.vllm.ai/en/latest/getting_started/examples/basic.html) - [AutoAWQ documentation](https://docs.vllm.ai/en/latest/features/quantization/auto_awq.html) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 13, 2025, 9:04am UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/9 "2025-08-13T09:04:05Z")
In this example, the 256GB offload, is it the maximum allocated? We would like vllm to use VRAM first then use whatever is needed from RAM, maybe not the complete 256GB ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 13, 2025, 9:04am UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/10 "2025-08-13T09:04:33Z")
Yes, `--cpu-offload-gb 256` sets the maximum CPU RAM vLLM can use for offloading, but vLLM will always fill GPU VRAM first and only use as much RAM as needed to fit the model weights that exceed VRAM. If the model fits in VRAM, RAM offload is not used; if not, only the required portion of the specified RAM is used, up to the limit you set. You do not need to allocate the full 256GB unless the model requires it—this is just an upper bound. Would you like more detail or a usage example? Sources: - [vLLM Basic Example: CPU offload](https://docs.vllm.ai/en/latest/getting_started/examples/basic.html) - [vLLM GitHub Issue: Offload Model Weights to CPU](https://github.com/vllm-project/vllm/issues/3563) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 13, 2025, 4:43pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/11 "2025-08-13T16:43:19Z")
I have implemented this: args: - “–model” - “QuantTrio/Qwen3-Coder-480B-A35B-Instruct-AWQ” - “–tensor-parallel-size” - “4” - “–max-model-len” # Reduce for testing/memory efficiency - “4096” # Start with a more manageable sequence length, e.g., 4096 or 8192 - “–quantization” - “awq\_marlin” - “–cpu-offload-gb” - “512” - “–trust-remote-code” but I still get error torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 600.00 MiB. GPU 0 has a total capacity of 44.40 GiB of which 361.00 MiB is free. Including non-PyTorch memory, this process has 42.76 GiB memory in use. Of the allocated memory 42.03 GiB is allocated by PyTorch, and 145.79 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True to avoid fragmentation ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 13, 2025, 4:44pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/12 "2025-08-13T16:44:08Z")
Even with AWQ quantization and `--cpu-offload-gb`, vLLM still requires enough VRAM to hold part of the model and the KV cache. If the model and cache together exceed available VRAM, you will get CUDA OOM errors. The `--cpu-offload-gb` only offloads model weights, not the KV cache, which must fit in VRAM. For extremely large models like Qwen3-Coder-480B, even with quantization and offload, 44GB VRAM per GPU may not be enough, especially with a sequence length of 4096 and 4-way tensor parallelism. See [AutoAWQ documentation](https://docs.vllm.ai/en/latest/features/quantization/auto_awq.html) and [CPU offload usage](https://docs.vllm.ai/en/latest/getting_started/examples/basic.html). Would you like more detail on how to further reduce memory usage or debug this? Sources: - [https://github.com/vllm-project/vllm/issues/15598](https://github.com/vllm-project/vllm/issues/15598) - [https://github.com/vllm-project/vllm/issues/8721](https://github.com/vllm-project/vllm/issues/8721) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 13, 2025, 4:46pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/13 "2025-08-13T16:46:15Z")
How can I reduce memory usage in this case? Which parameters to modify? It will be for 1 user, 1 batch. Should I set this PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True too? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 13, 2025, 4:46pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/14 "2025-08-13T16:46:53Z")
To reduce memory usage for a single user and batch, lower `--max-model-len` (e.g., 1024 or 2048), set `--max-num-seqs 1`, and reduce `--gpu-memory-utilization` (e.g., 0.7). These changes shrink the KV cache and overall VRAM usage. Also, setting `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` can help avoid fragmentation and may prevent OOM errors, as suggested in [multiple vLLM issues](https://github.com/vllm-project/vllm/issues/15598). Example args: ```yaml args: - "--max-model-len" - "1024" - "--max-num-seqs" - "1" - "--gpu-memory-utilization" - "0.7" ``` And set the environment variable in your pod: ```yaml env: - name: PYTORCH_CUDA_ALLOC_CONF value: "expandable_segments:True" ``` Would you like more detail or a full manifest example? Sources: - [vLLM Issue: CUDA OOM and memory reduction](https://github.com/vllm-project/vllm/issues/15598) - [vLLM Conserving Memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory.html) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 13, 2025, 4:50pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/15 "2025-08-13T16:50:34Z")
> [@RunLLM](#): > > ```auto > - "--max-num-seqs" > - "1" > > ``` what is the default value of - “–max-num-seqs” ? If I increase it to 5 will it have huge impact to the required memory ? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 13, 2025, 4:51pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/16 "2025-08-13T16:51:08Z")
The default value of `--max-num-seqs` in vLLM V1 is 1024, but it may be set to 256 in some recent versions or specific configurations. Increasing `--max-num-seqs` from 1 to 5 will increase memory usage, as it controls the maximum number of concurrent sequences (batches) and directly affects the size of the KV cache in VRAM. The increase is roughly linear, so memory usage for 5 will be about 5 times that for 1, but still much less than the default of 1024 or 256. See [Optimization and Tuning](https://docs.vllm.ai/en/latest/configuration/optimization.html) and [Conserving Memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory.html). Would you like more detail on how to estimate the exact memory usage? Sources: - [Optimization and Tuning](https://docs.vllm.ai/en/latest/configuration/optimization.html) - [Conserving Memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory.html) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 13, 2025, 10:26pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/17 "2025-08-13T22:26:22Z")
The LLMs deployed with vLLM (quantized or not), do they have tooling enabled by default or we need to pass some parameters in Kubernetes manifest for this? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 13, 2025, 10:26pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/18 "2025-08-13T22:26:44Z")
Tool calling is not enabled by default in vLLM. You must explicitly enable it by passing `--enable-auto-tool-choice` and `--tool-call-parser ` as arguments in your Kubernetes manifest. The parser name depends on your model (e.g., `hermes`, `llama`, `mistral`, etc.). See [Tool Calling documentation](https://docs.vllm.ai/en/stable/features/tool_calling.html) for details. Would you like a manifest example or more information on supported parsers? Sources: - [vLLM GitHub Issue: How to enable tool calling](https://github.com/vllm-project/vllm/issues/19601) ---
**Author:** ![anticool](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@anticool](https://discuss.vllm.ai/u/anticool)\ **Post date:** [August 13, 2025, 10:30pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/19 "2025-08-13T22:30:42Z")
For Qwen models, do I just pass those 2 parameters --tool-call-parser hermes and enable-auto-tool-choice? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 13, 2025, 10:30pm UTC](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354/20 "2025-08-13T22:30:59Z")
Yes, for Qwen models, you just need to pass `--enable-auto-tool-choice` and `--tool-call-parser hermes` to enable tool calling. The Qwen tokenizer config already supports Hermes-style tool use, so no extra chat template is required. See [Qwen Models Tool Calling](https://docs.vllm.ai/en/stable/features/tool_calling.html#qwen-models). Would you like more detail or a manifest example? Sources: - [Qwen Models Tool Calling](https://docs.vllm.ai/en/stable/features/tool_calling.html#qwen-models) [Next page](https://discuss.vllm.ai/t/deploy-a-big-llm-when-gpu-vram-not-enough/1354.md?page=2)