# Why vllm cannot fully use GPU in batch processing **URL:** \ **Category:** General\ **Created:** [March 28, 2025, 10:36am UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305 "2025-03-28T10:36:36Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![bean](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bean](https://discuss.vllm.ai/u/bean)\ **Post date:** [March 28, 2025, 10:36am UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/1 "2025-03-28T10:36:36Z")
I’m running with vllm to process a large dataset for inference. I have tried several parameters, and found that they do not work in my case. The GPU utilization is very low (10% utilization with nearly no GPU power increase) ```auto self.llm = LLM( model=self.model_name, trust_remote_code=True, # max_model_len=4096, # max_num_seqs=512*10, # max_num_batched_tokens=512*10, # enable_chunked_prefill=True, # enable_prefix_caching=True, ) llm.chat(prompts, sampling_params) ``` prompts can be very large (e.g., list from 1000 to 100000) Thank you for your help! ---
**Author:** ![dingfen](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/dingfen/32/117_2.png) [@dingfen](https://discuss.vllm.ai/u/dingfen)\ **Post date:** [March 28, 2025, 12:58pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/2 "2025-03-28T12:58:19Z")
maybe you could try: self.llm.generate(prompts, sampling\_params) ---
**Author:** ![chris](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/chris/32/119_2.png) [@chris](https://discuss.vllm.ai/u/chris)\ **Post date:** [March 28, 2025, 2:40pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/3 "2025-03-28T14:40:58Z")
If your GPU is Nvidia, use this Python to check whether’s it’s available for VLLM to use. ```auto import torch if torch.cuda.is_available(): print(f"GPU {torch.cuda.get_device_name(0)} looking good") else: print("GPU is not available. Will use CPU.") ``` You may already know that, but because this is the first post about why vllm cannot use GPU, this answer can be here for folks to find. Note: Your GPU being available as reported here is no guarantee anything will work. Only the converse is true (for Nvidia specifically wrt to above Python). “No GPU available” means VLLM will use your CPU exclusively. (unless i am wrong - i am very new at this) ---
**Author:** ![comaniac](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/comaniac/32/5_2.png) [@comaniac](https://discuss.vllm.ai/u/comaniac)\ **Post date:** [March 28, 2025, 3:46pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/4 "2025-03-28T15:46:13Z")
Can you provide more information such as the model and GPU you’re using, and how many prompts you are processing in total? ---
**Author:** ![bean](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bean](https://discuss.vllm.ai/u/bean)\ **Post date:** [March 28, 2025, 4:06pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/5 "2025-03-28T16:06:54Z")
Is there any difference between these? I see the doc said chat is more easy to use, and it calls generate ---
**Author:** ![bean](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bean](https://discuss.vllm.ai/u/bean)\ **Post date:** [March 28, 2025, 4:07pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/6 "2025-03-28T16:07:52Z")
It use gpu, I can see it have some utilization, but not high as I expected ---
**Author:** ![bean](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bean](https://discuss.vllm.ai/u/bean)\ **Post date:** [March 28, 2025, 4:08pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/7 "2025-03-28T16:08:22Z")
I use A100 GPU, and about 10,000 prompts to process. ---
**Author:** ![bean](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bean](https://discuss.vllm.ai/u/bean)\ **Post date:** [March 28, 2025, 4:09pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/8 "2025-03-28T16:09:29Z")
After some investigations I found if the batch size is too large it works not good. If I reduce it to reasonable size it can get better utilization. But still just about 30% ---
**Author:** ![comaniac](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/comaniac/32/5_2.png) [@comaniac](https://discuss.vllm.ai/u/comaniac)\ **Post date:** [March 28, 2025, 4:29pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/9 "2025-03-28T16:29:34Z")
Oh yeah this is sort of a well-known issue. I’d suggest either of the following: 1. Batching by yourself. This is a straightforward approach to make it work, but you’ll still see under utilization between batches, because in this case we have to 100% finish a batch before start processing the next batch. ```auto for batch in dataset: outputs += llm.chat(batch, sampling_params) ``` 1. Continuous batching with async (advance): ---
**Author:** ![bean](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bean](https://discuss.vllm.ai/u/bean)\ **Post date:** [March 29, 2025, 3:08am UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/10 "2025-03-29T03:08:00Z")
Thank you for your suggestions. It seems reasonable. I have an additional question. It seems I can use a very large batch in the first method (absolutely exceeding the GPU memory). How will vllm process this? It will also raise a queue and use continuous batching? ---
**Author:** ![comaniac](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/comaniac/32/5_2.png) [@comaniac](https://discuss.vllm.ai/u/comaniac)\ **Post date:** [March 29, 2025, 3:38am UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/11 "2025-03-29T03:38:09Z")
Yes vllm internally also has a queue for all requests, and uses continuous batching to batch requests, but it seems like the scalability isn’t that good… ---
**Author:** ![bean](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bean](https://discuss.vllm.ai/u/bean)\ **Post date:** [March 29, 2025, 4:18am UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/12 "2025-03-29T04:18:40Z")
Yeah, so may be a easy solution is to use vllm serve, and then another process to asyncio send request to this service? (somewhat like your second solution) ---
**Author:** ![chris](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/chris/32/119_2.png) [@chris](https://discuss.vllm.ai/u/chris)\ **Post date:** [March 29, 2025, 12:59pm UTC](https://discuss.vllm.ai/t/why-vllm-cannot-fully-use-gpu-in-batch-processing/305/13 "2025-03-29T12:59:45Z")
Right - Your question sounded like you know more about vllm than me (as everyone who knows anything about vllm does). I wrote my answer for search engines crawling for vllm+use+GPU. Adding to my answer for other n00bs to come: cuda\_12.8.x\_570.x.x.run runfile installation takes unticked checkboxes to mean “UNinstall” rather than “Do not install” as the label says. If you re-run the cuda install runfile to, say, repair your cuda toolkit that you might have messed up somehow, and you already have the nvidia gpu driver installed, you cannot use runfile to update/reinstall cuda toolkit items unless you go to runlevel 3 and reinstall the nvidia driver too. Otherwise you end up with your cuda toolkit reinstalled, but your GPU driver got uninstalled! Maybe I am old and an unticked “install” box changed meaning from “don’t install” to “UNinstall if it’s already present” and everybody knows this, but for any other old folks like me: “Don’t install” seems to mean “Uninstall” to Nvidia (at least), so watch out.