The vLLM memory allocation for discrete and integrated GPUs needs to be planed differently. For dGPU, the default GPU memory utilization is set to 0.8 (80%) of the total GPU memory. This may not suitable for iGPU in which the memory is shared with operating system, applications, display memory, and vLLM own use.
Following is the iGPU system I use for testing.
- Hardware Specification:
CPU: Intel® Core™ Ultra X7 Processor 358H
CPU Cores: 16 (4 Performance-cores, 8 Efficient-cores, and 4 Low Power Efficient-cores)
CPU Threads: 16
Memory: 32GB
iGPU: Intel® Arc™ B390 GPU - Software Version
Ubuntu 26.04 LTS
Intel Graphics Compute Runtime 26.22.38646.4
Docker 29.3.1
vLLM commit 3ca6ca2
And the following is an example on the memory planning.
- Operating system and application are already utilizing 5GB and could be more.
- iGPU can use up to 28GB. But it has only 18GB free (display has used 9GB). So, the default vLLM setting of 80% (0.8 x 28GB = 22.5GB) allocation is already not feasible. For 6GB model, I set it to 30% (0.3 x 28GB = 8.4GB). The extra 2.4GB is still not enough for the default 250K or 1M model token size. So, I set the maximum to 4096 input/output tokens. Please note that vLLM can run requests in parallel, therefore the 2.4GB can accommodate about 17x requests with the 4096 tokens (1GB can store 29,120 tokens).
- Other vLLM own memory like KV cache is set to 4GB as the default. In the following example, I will reduce it to 1GB. It is enough for 29,120 tokens.
So, the total is 23.4GB (5GB for OS and application, 9GB for display, 6GB for the model, 2.4GB for input/output tokens, and 1GB for KV cache). There are other memory requirements for multi-modality. Please refer to vLLM Conserving Memory document for more details.
Please build the vLLM Docker image to run on Intel Arc GPU using following commands.
$ git clone https://github.com/vllm-project/vllm.git
$ cd vllm
$ git checkout 3ca6ca2
$ sudo docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .
$ sudo docker images
IMAGE ID DISK USAGE CONTENT SIZE
vllm-xpu-env:latest b77c4d963d71 37.8GB 9.23GB
Serve the vLLM with the 23.4GB budget we have planned.
$ vllm serve \
Qwen/Qwen2.5-3B-Instruct \
--dtype=bfloat16 \
--tensor-parallel-size 1 \
--enforce-eager \
--attention-backend TRITON_ATTN \
--gpu-memory-utilization 0.3 \
--max-model-len 4096 \
--kv-cache-memory-bytes 1G
(EngineCore pid=538) INFO 07-20 06:45:06 [default_loader.py:391] Loading weights took 0.72 seconds
(EngineCore pid=538) INFO 07-20 06:45:06 [gpu_model_runner.py:4883] Model loading took 5.79 GiB memory and 121.401320 seconds
(EngineCore pid=538) INFO 07-20 06:45:28 [gpu_worker.py:381] Initial free memory 16.06 GiB, reserved 1.0 GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling. This does not respect the gpu_memory_utilization config. Only use kv_cache_memory_bytes config when you want manual control of KV cache memory size. If OOM'ed, check the difference of initial free memory between the current run and the previous run where kv_cache_memory_bytes is suggested and update it correspondingly.
(EngineCore pid=538) INFO 07-20 06:45:28 [kv_cache_utils.py:1710] GPU KV cache size: 29,120 tokens
(EngineCore pid=538) INFO 07-20 06:45:28 [kv_cache_utils.py:1711] Maximum concurrency for 4,096 tokens per request: 7.11x
Test inference.
curl -X POST "http://localhost:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "Qwen/Qwen2.5-3B-Instruct",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Write fibonacci code in python. And explain."
}
]
}
]
}'
{"id":"chatcmpl-b6645c5657b60e2a","object":"chat.completion","created":1784530149,"model":"Qwen/Qwen2.5-3B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Certainly! Below is a simple Python program to generate the Fibonacci sequence up to a certain number of terms. I'll provide both a basic version and a more efficient version using memoization (dynamic programming) to handle larger sequences.\n\n### Basic Fibonacci Sequence\n\n```python\ndef fibonacci_basic(n):\n if n <= 0:\n return []\n elif n == 1:\n return [0]\n elif n == 2:\n return [0, 1]\n\n fib_sequence = [0, 1]\n for i in range(2, n):\n next_value = fib_sequence[-1] + fib_sequence[-2]\n fib_sequence.append(next_value)\n \n return fib_sequence\n\n# Example usage:\nn = 10\nprint(fibonacci_basic(n))\n```\n\n### Efficient Fibonacci Sequence Using Memoization\n\nFor generating large Fibonacci numbers, it's more efficient to use memoization to avoid redundant calculations.\n\n```python\ndef fibonacci_memo(n, memo={}):\n if n <= 0:\n return []\n elif n == 1:\n return [0]\n elif n == 2:\n return [0, 1]\n\n if n not in memo:\n memo[n] = fibonacci_memo(n-1, memo) + [fibonacci_memo(n-1, memo)[-1] + fibonacci_memo(n-2, memo)[-1]]\n \n return memo[n]\n\n# Example usage:\nn = 10\nprint(fibonacci_memo(n))\n```\n\nIn this example, `memo` is a dictionary that stores previously computed Fibonacci numbers to avoid recalculating them. The function checks if the value has already been computed and returns it directly from the dictionary if it exists.\n\nBoth functions will print the first `n` terms of the Fibonacci sequence. Choose the one that best fits your needs based on the size of `n`. For smaller values of `n`, the basic version should suffice, but for larger values or performance-critical applications, the memoized version is recommended.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":"vllm-0.20.1rc1.dev105+g3ca6ca210-db217ad7","usage":{"prompt_tokens":35,"total_tokens":452,"completion_tokens":417,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}(base)