# Multimodal inference guideline?

**URL:** <https://discuss.vllm.ai/t/multimodal-inference-guideline/698>\
**Category:** General\
**Created:** [May 27, 2025, 8:14am UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698 "2025-05-27T08:14:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [May 27, 2025, 8:14am UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/1 "2025-05-27T08:14:52Z")

</div>

Hi all.

Firsttime user, so i have some questions about multimodal inference. Below you can find some code of how i run it currently.

- As my card has 12 GB VRAM, shouldnt i be able to run eg a 7B model? I often get that there is not enough memory for KV-cache. Is there a guideline on which models require which amount of VRAM? Are there tricks to lower this amount?
- As i understand, the template of the prompt (when inputting an image) is rather important ,and changes per model. Is there an overview on which model uses which template?
- I often get output that is absolute nonsense? Sometimes i just get random chinese text/ text that doesnt pertain to the image/ text that stops mid-sentence/… . No clue what i’m doing wrong for this to happen.

Thanks for any help 🙂 !

```python
from vllm import LLM
import PIL.Image
import multiprocessing as mp

mp.set_start_method("spawn", force=True)

def main():
    # Initialize the model
    llm = LLM(
        model="OpenGVLab/InternVL2-1B",
        gpu_memory_utilization=0.9,
        enforce_eager=True,
    )

    # Prepare the multimodal prompt with the correct placeholder
    prompt = "What is the content of this image in English?\n<image>"

    # Load image
    image = PIL.Image.open("./image.png")

    # Generate
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image},
    })

    # Print results
    for o in outputs:
        print(o.outputs[0].text)

if __name__ == " __main__":
    main()

```

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 27, 2025, 8:15am UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/2 "2025-05-27T08:15:27Z")

</div>

You can run 7B models on a 12GB GPU, but only with careful memory management—reduce `max_model_len`, batch size (`max_num_seqs`), and set `gpu_memory_utilization` below 1.0. Also, disabling CUDA graphs with `enforce_eager=True` helps lower memory usage, but may reduce performance. See [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md) and [context length and batch size](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size) for details.

Prompt templates are model-specific and must match the format expected by each model (e.g., `<image>`, `<|image_1|>`, etc.). Refer to the HuggingFace model card or vLLM’s [vision-language example](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py) for the correct template. Incorrect templates or tokenization can cause nonsense or incomplete outputs. For InternVL2, use the chat template as shown in the [official example](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py).

Would you like more detail or a code example for a specific model?

Sources:

- [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)
- [context length and batch size](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [May 27, 2025, 8:48am UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/3 "2025-05-27T08:48:22Z")

</div>

The links for “official example” and “vision-language example” are not working. Can you point me to the correct examples? Also, give an overview of all the “tricks” for lowering memory usage.  
Also, is the code fully correct for multimodal image inference? Can you give example code?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 27, 2025, 8:48am UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/4 "2025-05-27T08:48:52Z")

</div>

The correct example for multimodal image inference is [vision\_language.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py) and for multi-image input, see [vision\_language\_multi\_image.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language_multi_image.py). For InternVL2, use the chat template as shown in these examples and the [multimodal inputs documentation](https://docs.vllm.ai/en/stable/serving/multimodal_inputs.html).

To lower memory usage:

- Reduce `max_model_len` and `max_num_seqs` ([docs](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size))
- Set `gpu_memory_utilization` below 1.0 ([docs](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md))
- Use quantized models if available
- Use tensor parallelism if you have multiple GPUs
- Limit the number of images per prompt with `limit_mm_per_prompt`
- Adjust multi-modal processor arguments (e.g., `mm_processor_kwargs`)
- Disable CUDA graphs with `enforce_eager=True` ([docs](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md))

Here is a correct example for single-image inference with InternVL2:

```python
from vllm import LLM
from PIL import Image

llm = LLM(
    model="OpenGVLab/InternVL2-1B",
    gpu_memory_utilization=0.9,
    enforce_eager=True,
    limit_mm_per_prompt={"image": 1},
)

prompt = "<|im_start|>User\n<image>\nWhat is the content of this image in English?\n<|im_end|>\n<|im_start|>Assistant\n"
image = Image.open("./image.png")

outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {"image": image},
})

for o in outputs:
    print(o.outputs[0].text)

```

Would you like more detail on any of these points?

Sources:

- [vision\_language\_multi\_image.py](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language_multi_image.py)
- [multimodal inputs documentation](https://docs.vllm.ai/en/stable/serving/multimodal_inputs.html)
- [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)
- [context length and batch size](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [May 27, 2025, 1:19pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/5 "2025-05-27T13:19:06Z")

</div>

Give me the correct prompt templates (in code) for all the following model families:

- internVL
- Qwen
- deepseek VL
- gemma
- minicpm
- llama
- llava

One other problem i have is that the gemma models seem to be locked? i cant use them withouth agreeing to certain things?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 27, 2025, 1:19pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/6 "2025-05-27T13:19:31Z")

</div>

It could be possible that the correct prompt templates for each model family are as follows, based on official vLLM and HuggingFace examples. Note that prompt format is critical and must match the model’s expected input. For Gemma models, it might be that you need to accept a license on HuggingFace before downloading—this is a restriction from the model provider, not vLLM.

**Prompt templates (Python code snippets):**

- **InternVL:**

```python
messages = [[{'role': 'user', 'content': f"<image>\n{question}"}] for question in questions]
prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

```

- **Qwen:**

```python
messages = [[{'role': 'user', 'content': [{'type': 'image'}, {'type': 'text', 'text': question}]}] for question in questions]
prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

```

- **Deepseek VL:**

```python
prompts = [f"<|User|>: <image>\n{question}\n\n<|Assistant|>:" for question in questions]

```

- **Gemma:**

```python
prompts = [("<bos><start_of_turn>user\n"
            f"<start_of_image>{question}<end_of_turn>\n"
            "<start_of_turn>model\n") for question in questions]

```

- **MiniCPM:**

```python
prompts = [f"<image>{question}" for question in questions]

```

- **Llama (vision):**

```python
prompts = [f"USER: <image>\n{question}\nASSISTANT:" for question in questions]

```

- **LLaVA:**

```python
prompts = [f"USER: <image>\n{question}\nASSISTANT:" for question in questions]

```

For Gemma models, you must agree to the license on HuggingFace before downloading—this is not a vLLM limitation ([source](https://github.com/vllm-project/vllm/pull/14660)). Would you like more detail or code for a specific model?

Sources:

- [https://github.com/vllm-project/vllm/pull/14660](https://github.com/vllm-project/vllm/pull/14660)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [May 27, 2025, 1:29pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/7 "2025-05-27T13:29:51Z")

</div>

When processing the image of a white kitten in a field, the output is drastically cut off? How can this be?

/home/bdck/PROJECTS\_WSL/VLM/venv\_vlm/bin/python /home/bdck/PROJECTS\_WSL/VLM/VLM\_redoUsingVllm.py  
INFO 05-27 15:28:28 [**init**.py:239] Automatically detected platform cuda.  
INFO 05-27 15:28:35 [config.py:600] This model supports multiple tasks: {‘classify’, ‘embed’, ‘score’, ‘reward’, ‘generate’}. Defaulting to ‘generate’.  
INFO 05-27 15:28:35 [config.py:1780] Chunked prefill is enabled with max\_num\_batched\_tokens=8192.  
WARNING 05-27 15:28:35 [cuda.py:96] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used  
WARNING 05-27 15:28:36 [utils.py:2273] We must use the `spawn` multiprocessing start method. Overriding VLLM\_WORKER\_MULTIPROC\_METHOD to ‘spawn’. See [Troubleshooting - vLLM](https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing) for more information. Reason: CUDA is initialized  
INFO 05-27 15:28:39 [**init**.py:239] Automatically detected platform cuda.  
INFO 05-27 15:28:40 [core.py:61] Initializing a V1 LLM engine (v0.8.3) with config: model=‘OpenGVLab/InternVL3-2B’, speculative\_config=None, tokenizer=‘OpenGVLab/InternVL3-2B’, skip\_tokenizer\_init=False, tokenizer\_mode=auto, revision=None, override\_neuron\_config=None, tokenizer\_revision=None, trust\_remote\_code=False, dtype=torch.bfloat16, max\_seq\_len=512, download\_dir=None, load\_format=auto, tensor\_parallel\_size=1, pipeline\_parallel\_size=1, disable\_custom\_all\_reduce=False, quantization=None, enforce\_eager=True, kv\_cache\_dtype=auto, device\_config=cuda, decoding\_config=DecodingConfig(guided\_decoding\_backend=‘xgrammar’, reasoning\_backend=None), observability\_config=ObservabilityConfig(show\_hidden\_metrics=False, otlp\_traces\_endpoint=None, collect\_model\_forward\_time=False, collect\_model\_execute\_time=False), seed=None, served\_model\_name=OpenGVLab/InternVL3-2B, num\_scheduler\_steps=1, multi\_step\_stream\_outputs=True, enable\_prefix\_caching=True, chunked\_prefill\_enabled=True, use\_async\_output\_proc=False, disable\_mm\_preprocessor\_cache=False, mm\_processor\_kwargs=None, pooler\_config=None, compilation\_config={“splitting\_ops”:,“compile\_sizes”:,“cudagraph\_capture\_sizes”:,“max\_capture\_size”:0}  
2025-05-27 15:28:40,701 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend  
WARNING 05-27 15:28:40 [utils.py:2413] Methods determine\_num\_available\_blocks,device\_config,get\_cache\_block\_size\_bytes,initialize\_cache not implemented in \<vllm.v1.worker.gpu\_worker.Worker object at 0x7f07415a1990\>  
INFO 05-27 15:28:41 [parallel\_state.py:957] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0  
WARNING 05-27 15:28:41 [interface.py:304] Using ‘pin\_memory=False’ as WSL is detected. This may slow down the performance.  
INFO 05-27 15:28:41 [cuda.py:221] Using Flash Attention backend on V1 engine.  
INFO 05-27 15:28:41 [gpu\_model\_runner.py:1258] Starting to load model OpenGVLab/InternVL3-2B…  
INFO 05-27 15:28:41 [config.py:3334] cudagraph sizes specified by model runner is overridden by config   
INFO 05-27 15:28:42 [topk\_topp\_sampler.py:44] Currently, FlashInfer top-p & top-k sampling sampler is disabled because FlashInfer\>=v0.2.3 is not backward compatible. Falling back to the PyTorch-native implementation of top-p & top-k sampling.  
INFO 05-27 15:28:42 [weight\_utils.py:265] Using model weights format [‘\*.safetensors’]  
INFO 05-27 15:28:42 [weight\_utils.py:315] No model.safetensors.index.json found in remote.  
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00\<?, ?it/s]  
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00\<00:00, 1.70it/s]  
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00\<00:00, 1.70it/s]

INFO 05-27 15:28:43 [loader.py:447] Loading weights took 0.62 seconds  
INFO 05-27 15:28:43 [gpu\_model\_runner.py:1273] Model loading took 3.9110 GiB and 1.253380 seconds  
INFO 05-27 15:28:43 [gpu\_model\_runner.py:1542] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 3 image items of the maximum feature size.  
INFO 05-27 15:28:45 [kv\_cache\_utils.py:578] GPU KV cache size: 18,528 tokens  
INFO 05-27 15:28:45 [kv\_cache\_utils.py:581] Maximum concurrency for 512 tokens per request: 36.19x  
INFO 05-27 15:28:45 [core.py:162] init engine (profile, create kv cache, warmup model) took 2.03 seconds

The content of this image is in English. It shows a picture of a small  
[rank0]:[W527 15:28:46.848639031 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy\_process\_group() was not called before program exit, which can leak resources. For more info, please see [Distributed communication package - torch.distributed — PyTorch 2.7 documentation](https://pytorch.org/docs/stable/distributed.html#shutdown) (function operator())

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 27, 2025, 1:30pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/8 "2025-05-27T13:30:27Z")

</div>

Your output is cut off because your `max_seq_len` (context length) is set to 512, which is very short for multimodal models—image tokens plus text tokens can easily exceed this, causing the model to truncate or stop mid-sentence. This is a common cause of incomplete outputs, especially for vision models where the image embedding consumes many tokens ([source](https://github.com/vllm-project/vllm/issues/12165)).

To fix this, increase `max_seq_len` (or `max_model_len`) to the largest value your GPU can handle (e.g., 1024, 2048, or higher if memory allows). If you still hit memory errors, reduce batch size (`max_num_seqs=1`), use quantized models, or lower image resolution via `mm_processor_kwargs`. See [context length and batch size](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size) and [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md) for more.

Would you like a code example for setting these parameters?

Sources:

- [https://github.com/vllm-project/vllm/issues/12165](https://github.com/vllm-project/vllm/issues/12165)
- [context length and batch size](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size)
- [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [May 27, 2025, 1:38pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/9 "2025-05-27T13:38:00Z")

</div>

With nothing changed, i set max\_model\_len = 30000 , which should be more than enough to get a nice input+output. However the only thing i get is still cut off:

The image depicts a young kitten walking on a grassy field. The kitten is  
[rank0]:[W527 15:36:42.900602291 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy\_process\_group() was not called before program exit, which can leak resources. For more info, please see [Distributed communication package - torch.distributed — PyTorch 2.7 documentation](https://pytorch.org/docs/stable/distributed.html#shutdown) (function operator())

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 27, 2025, 1:38pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/10 "2025-05-27T13:38:27Z")

</div>

If you set `max_model_len=30000` but still get cut-off output, it’s likely your GPU does not have enough memory to allocate a KV cache for such a long context, so vLLM silently reduces the effective context length or fails to allocate enough cache ([see here](https://github.com/vllm-project/vllm/issues/7526)). You can check the logs for lines like “The model’s max seq len (30000) is larger than the maximum number of tokens that can be stored in KV cache (…)” or similar warnings.

To resolve this, set `max_model_len` to a value that fits your GPU memory (e.g., 1024–4096 for 12GB VRAM), and check the actual allocated KV cache size in the logs. Also, reduce `max_num_seqs` to 1 and lower image resolution with `mm_processor_kwargs` if needed ([conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)). Would you like a code example for these settings?

Sources:

- [https://github.com/vllm-project/vllm/issues/7526](https://github.com/vllm-project/vllm/issues/7526)
- [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [May 27, 2025, 1:48pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/11 "2025-05-27T13:48:08Z")

</div>

INFO 05-27 15:44:22 [loader.py:447] Loading weights took 0.40 seconds  
INFO 05-27 15:44:22 [gpu\_model\_runner.py:1273] Model loading took 1.7627 GiB and 42.797461 seconds  
INFO 05-27 15:44:22 [gpu\_model\_runner.py:1542] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 3 image items of the maximum feature size.  
INFO 05-27 15:44:24 [kv\_cache\_utils.py:578] GPU KV cache size: 296,016 tokens  
INFO 05-27 15:44:24 [kv\_cache\_utils.py:581] Maximum concurrency for 4,096 tokens per request: 72.27x  
INFO 05-27 15:44:24 [core.py:162] init engine (profile, create kv cache, warmup model) took 1.79 seconds  
The image showcases a cute, small kitten with a gray and white coat running on

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 27, 2025, 1:48pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/12 "2025-05-27T13:48:53Z")

</div>

Even though your GPU KV cache size is 296,016 tokens, your prompt and output are still cut off because the model’s `max_model_len` or the prompt template may be limiting the actual output length, or the model’s stopping criteria are being triggered early. Also, if you set a low `max_tokens` in your sampling parameters, the output will be short by default ([see here](https://github.com/vllm-project/vllm/issues/5395)).

To get longer outputs, explicitly set a higher `max_tokens` in your sampling parameters (e.g., `max_tokens=200`). Also, ensure your prompt template is correct for the model. Would you like a code example for this?

Sources:

- [https://github.com/vllm-project/vllm/issues/5395](https://github.com/vllm-project/vllm/issues/5395)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [May 27, 2025, 1:50pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/13 "2025-05-27T13:50:56Z")

</div>

with all the info from above, create the best coding example. Using the correct params, etc etc

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 27, 2025, 1:51pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/14 "2025-05-27T13:51:47Z")

</div>

Here is a best-practice example for multimodal inference with vLLM, using InternVL2-1B, with all recommended parameters for a 12GB GPU. This code sets a reasonable context length, batch size, and sampling parameters, and uses the correct prompt template for InternVL. Adjust `max_model_len` and image path as needed.

```python
from vllm import LLM, SamplingParams
from PIL import Image
from transformers import AutoTokenizer

# Set up model and tokenizer
model_name = "OpenGVLab/InternVL2-1B"
llm = LLM(
    model=model_name,
    gpu_memory_utilization=0.9, # Use 90% of GPU memory
    enforce_eager=True, # Lower memory usage (slower)
    max_model_len=2048, # Fits 12GB VRAM, adjust if needed
    max_num_seqs=1, # Single request at a time
    limit_mm_per_prompt={"image": 1} # Only 1 image per prompt
)

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

# Prepare prompt using the correct template
question = "What is the content of this image in English?"
messages = [{"role": "user", "content": f"<image>\n{question}"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Load image
image = Image.open("./image.png").convert("RGB")

# Set sampling parameters for longer, deterministic output
sampling_params = SamplingParams(
    temperature=0.0, # Deterministic output
    max_tokens=200, # Allow up to 200 output tokens
    top_p=1.0,
    top_k=1
)

# Run inference
outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {"image": image},
}, sampling_params=sampling_params)

# Print result
for o in outputs:
    print(o.outputs[0].text)

```

This example ensures correct prompt formatting, memory usage, and output length. For other models, adjust the prompt template as described previously.

Sources:

- [vision\_language.py example](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py)
- [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)
- [context length and batch size](https://docs.vllm.ai/en/stable/serving/offline_inference.html#context-length-and-batch-size)
- [multimodal inputs documentation](https://docs.vllm.ai/en/stable/serving/multimodal_inputs.html)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [June 20, 2025, 9:43am UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/15 "2025-06-20T09:43:10Z")

</div>

How do i silence the print output of the vllm? For each image i run on i get a tqdm output, which i dont want

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 20, 2025, 9:43am UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/16 "2025-06-20T09:43:33Z")

</div>

To silence the tqdm progress bar output in vLLM, set `use_tqdm=False` in the `generate` (or `encode`/`chat`) method call. For example:

```python
outputs = llm.generate({...}, sampling_params=sampling_params, use_tqdm=False)

```

This disables the tqdm bar for each run ([source](https://github.com/vllm-project/vllm/issues/6660), [vision\_language.py example](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py)). Would you like more detail on silencing other logs?

Sources:

- [https://github.com/vllm-project/vllm/issues/6660](https://github.com/vllm-project/vllm/issues/6660)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [June 23, 2025, 1:31pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/17 "2025-06-23T13:31:19Z")

</div>

when trying to inference a deepseek-VL2 model i get the following error, even though i use the same inferencing code as other models:

INFO 06-23 15:29:50 [**init**.py:239] Automatically detected platform cuda.  
Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you’ve just configured.  
WARNING:huggingface\_hub.\_login:Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you’ve just configured.

Evaluating deepseek-ai/deepseek-vl2…  
WARNING 06-23 15:29:52 [registry.py:429] No model architectures are specified  
Failed to evaluate deepseek-ai/deepseek-vl2: ‘NoneType’ object is not iterable

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 23, 2025, 1:32pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/18 "2025-06-23T13:32:34Z")

</div>

This error occurs because the DeepSeek-VL2 model’s `config.json` is missing the `architectures` field, which vLLM requires to resolve the model implementation. You must explicitly set the architecture using the `hf_overrides` parameter:

```python
llm = LLM(
    model="deepseek-ai/deepseek-vl2",
    hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]},
    ...
)

```

This is documented in the [supported models page](https://docs.vllm.ai/en/latest/models/supported_models.html#id2) and discussed in [this issue](https://github.com/vllm-project/vllm/issues/14105).

Would you like more detail or a full code example?

Sources:

- [https://github.com/vllm-project/vllm/issues/14105](https://github.com/vllm-project/vllm/issues/14105)

---

<div class="post-metadata">

**Author:** ![brechtBDCK](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/brechtbdck/32/328_2.png) [@brechtBDCK](https://discuss.vllm.ai/u/brechtBDCK)\
**Post date:** [June 24, 2025, 3:12pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/19 "2025-06-24T15:12:19Z")

</div>

how do i run my code on multiple gpus for maximal VRAM? I have two 24 GB VRAM gpus

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 24, 2025, 3:12pm UTC](https://discuss.vllm.ai/t/multimodal-inference-guideline/698/20 "2025-06-24T15:12:58Z")

</div>

To use both 24GB GPUs for maximal VRAM, set `tensor_parallel_size=2` when initializing the LLM, or use `--tensor-parallel-size 2` on the command line. This splits the model across both GPUs, allowing you to run larger models or longer context lengths ([distributed serving docs](https://docs.vllm.ai/en/latest/serving/distributed_serving.html), [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)).

Example in Python:

```python
from vllm import LLM
llm = LLM(model="your-model", tensor_parallel_size=2)

```

Or from the command line:

```bash
vllm serve your-model --tensor-parallel-size 2

```

Make sure both GPUs are visible (e.g., `CUDA_VISIBLE_DEVICES=0,1`). Would you like more detail on multi-GPU setup?

Sources:

- [distributed serving docs](https://docs.vllm.ai/en/latest/serving/distributed_serving.html)
- [conserving memory](https://github.com/vllm-project/vllm/blob/main/docs/configuration/conserving_memory.md)

[Next page](https://discuss.vllm.ai/t/multimodal-inference-guideline/698.md?page=2)