# Model Support **URL:** https://discuss.vllm.ai/c/model-support/10.md [Latest](https://discuss.vllm.ai/latest.md) · [Categories](https://discuss.vllm.ai/categories.md) --- ## [About the Model Support category](https://discuss.vllm.ai/t/about-the-model-support-category/76)
**Author:** [@youkaichao](https://discuss.vllm.ai/u/youkaichao)\ **Replies:** 0
--- ## [Mistral Small 4: ValueError: grammar is not of valid supported types. (StructuredOutputOptions.LARK)](https://discuss.vllm.ai/t/mistral-small-4-valueerror-grammar-is-not-of-valid-supported-types-structuredoutputoptions-lark/3024)
**Author:** [@jklinkenberg](https://discuss.vllm.ai/u/jklinkenberg)\ **Replies:** 1\ **Last updated:** [October 2, 2026, 12:32pm UTC](https://discuss.vllm.ai/t/mistral-small-4-valueerror-grammar-is-not-of-valid-supported-types-structuredoutputoptions-lark/3024 "2026-10-02T12:32:57Z")
We are discovering an issue with vLLM and Mistral Small 4 119B, which we host and provide for a larger user base. We tried to use regular vLLM Containers, however the generated and returned text was sometimes odd with s… --- ## [Does vLLM native KV Cache offloading to CPU work for gemm4?](https://discuss.vllm.ai/t/does-vllm-native-kv-cache-offloading-to-cpu-work-for-gemm4/2844)
**Author:** [@gushob21](https://discuss.vllm.ai/u/gushob21)\ **Replies:** 3\ **Last updated:** [July 28, 2026, 4:19pm UTC](https://discuss.vllm.ai/t/does-vllm-native-kv-cache-offloading-to-cpu-work-for-gemm4/2844 "2026-07-28T16:19:48Z")
I am using vllm/vllm-openai:gemma image to run gemma4-31b-it inference on G4 machine in Google Cloud. I am trying to offload the KV Cache to CPU by using this configuration: \`\`\` {“kv\_connector”: “OffloadingConnector”,“… --- ## [Mistral-small 4 - vLLM 0.23.0 : Problème avec la partie PIXTRAL](https://discuss.vllm.ai/t/mistral-small-4-vllm-0-23-0-probleme-avec-la-partie-pixtral/2729)
**Author:** [@Fred](https://discuss.vllm.ai/u/Fred)\ **Replies:** 1\ **Last updated:** [June 24, 2026, 8:29am UTC](https://discuss.vllm.ai/t/mistral-small-4-vllm-0-23-0-probleme-avec-la-partie-pixtral/2729 "2026-06-24T08:29:16Z")
Hello there! I’m using the Docker image vllm/vllm-openai:latest I’m currently using the mistralai/Mistral-Small-4-119B-2603 model and I’m having an issue with vLLM version 0.23.0. It shows an error when launching: Va… --- ## [How to run DeepSeek OCR 2 on vllm 0.23.0?](https://discuss.vllm.ai/t/how-to-run-deepseek-ocr-2-on-vllm-0-23-0/2727)
**Author:** [@dingfen](https://discuss.vllm.ai/u/dingfen)\ **Replies:** 1\ **Last updated:** [June 23, 2026, 10:40am UTC](https://discuss.vllm.ai/t/how-to-run-deepseek-ocr-2-on-vllm-0-23-0/2727 "2026-06-23T10:40:33Z")
I just tried to run DeepSeek OCR 2 on vllm 0.23.0 on H20, and here is my running command: vllm serve DeepSeek-OCR-2 --host 0.0.0.0 --port 8999 --tensor-parallel-size 2 --pipeline-parallel-size 1 --trust-remote-code --s… --- ## [Minimax m3 support](https://discuss.vllm.ai/t/minimax-m3-support/2689)
**Author:** [@DemircanOnur](https://discuss.vllm.ai/u/DemircanOnur)\ **Replies:** 1\ **Last updated:** [June 1, 2026, 7:32pm UTC](https://discuss.vllm.ai/t/minimax-m3-support/2689 "2026-06-01T19:32:27Z")
Hi! MiniMax just released M3 (API is live, open weights coming in ~10 days). The catch is that M3 moves away from full attention to a new mechanism they call MSA (MiniMax Sparse Attention) — block-level sparse selectio… --- ## [OOM Trying to run Gemma 4 31B NVFP4 on 2x16GB](https://discuss.vllm.ai/t/oom-trying-to-run-gemma-4-31b-nvfp4-on-2x16gb/2674)
**Author:** [@matthewlai](https://discuss.vllm.ai/u/matthewlai)\ **Replies:** 4\ **Last updated:** [May 31, 2026, 2:53pm UTC](https://discuss.vllm.ai/t/oom-trying-to-run-gemma-4-31b-nvfp4-on-2x16gb/2674 "2026-05-31T14:53:35Z")
Hello! I am trying to run Gemma 4 31B NVFP4 on 2x 5070 Ti (2x 16GB VRAM). With sharded model this should easily fit (and in fact it does run in llama.cpp with a 4-bit quant with plenty of VRAM left). However, I am getti… --- ## [GLM 5.1 PP support](https://discuss.vllm.ai/t/glm-5-1-pp-support/2634)
**Author:** [@liuyuhanalex](https://discuss.vllm.ai/u/liuyuhanalex)\ **Replies:** 1\ **Last updated:** [May 9, 2026, 10:20am UTC](https://discuss.vllm.ai/t/glm-5-1-pp-support/2634 "2026-05-09T10:20:50Z")
Hi vLLM team, I want to run GLM-5.1 with speculative decoding on 2 nodes, using TP + PP. My idea is: Run the main/target model with TP + PP across the 2 nodes. Keep the MTP draft model only on the last PP stage. Let … --- ## [How to extend the context length up to 1,010,000 tokens on Qwen3.5?](https://discuss.vllm.ai/t/how-to-extend-the-context-length-up-to-1-010-000-tokens-on-qwen3-5/2616)
**Author:** [@dingfen](https://discuss.vllm.ai/u/dingfen)\ **Replies:** 2\ **Last updated:** [May 4, 2026, 3:02pm UTC](https://discuss.vllm.ai/t/how-to-extend-the-context-length-up-to-1-010-000-tokens-on-qwen3-5/2616 "2026-05-04T15:02:49Z")
In normal cases, Qwen3.5 models ( Qwen3.5-397B for example) have Context Length: 262,144 natively. But Qwen/Qwen3.5-397B-A17B · Hugging Face also says it can be extended to at most 1,010,000 tokens. And I am curious abou… --- ## [The latest version of vllm is not compatible with local deployment of deepseek-v4（0.20）](https://discuss.vllm.ai/t/the-latest-version-of-vllm-is-not-compatible-with-local-deployment-of-deepseek-v4-0-20/2599)
**Author:** [@yiyanglu-pixel](https://discuss.vllm.ai/u/yiyanglu-pixel)\ **Replies:** 2\ **Last updated:** [April 29, 2026, 7:31pm UTC](https://discuss.vllm.ai/t/the-latest-version-of-vllm-is-not-compatible-with-local-deployment-of-deepseek-v4-0-20/2599 "2026-04-29T19:31:26Z")
I don’t use Docker and want to deploy deepseek-v4 locally using pip, but the latest version of vllm is 0.20. Do I need to wait for version 0.20.1 to be released? The GitHub source code compilation shows that the transfor… --- ## [Issues with Voxtral models and omni](https://discuss.vllm.ai/t/issues-with-voxtral-models-and-omni/2549)
**Author:** [@WittyExplorer](https://discuss.vllm.ai/u/WittyExplorer)\ **Replies:** 3\ **Last updated:** [April 14, 2026, 9:50pm UTC](https://discuss.vllm.ai/t/issues-with-voxtral-models-and-omni/2549 "2026-04-14T21:50:54Z")
I’m trying to setup voxtral models for STT and TTS. I have some issues and could use some advice. With \`mistralai/Voxtral-4B-TTS-2603\` I’m able to launch vllm omni but it doesn’t respect the memory limit. I use docker c… --- ## [Support for MiniMax-M2.5](https://discuss.vllm.ai/t/support-for-minimax-m2-5/2555)
**Author:** [@dingfen](https://discuss.vllm.ai/u/dingfen)\ **Replies:** 1\ **Last updated:** [April 14, 2026, 7:45am UTC](https://discuss.vllm.ai/t/support-for-minimax-m2-5/2555 "2026-04-14T07:45:36Z")
I am trying to run MiniMax-M2.5 on H200, and I have built a docker with vLLM 0.16.0 installed. How can I run MiniMax-M2.5 for best performance? --- ## [On 8-card Ascend 910B with vLLM serving Qwen3.5-122B-A10B, the client freezes at 8% progress when running accuracy test, as the server stops receiving new requests after Running reqs and KV Cache fall to 0.](https://discuss.vllm.ai/t/on-8-card-ascend-910b-with-vllm-serving-qwen3-5-122b-a10b-the-client-freezes-at-8-progress-when-running-accuracy-test-as-the-server-stops-receiving-new-requests-after-running-reqs-and-kv-cache-fall-to-0/2538)
**Author:** [@WangLedi](https://discuss.vllm.ai/u/WangLedi)\ **Replies:** 1\ **Last updated:** [April 11, 2026, 2:32am UTC](https://discuss.vllm.ai/t/on-8-card-ascend-910b-with-vllm-serving-qwen3-5-122b-a10b-the-client-freezes-at-8-progress-when-running-accuracy-test-as-the-server-stops-receiving-new-requests-after-running-reqs-and-kv-cache-fall-to-0/2538 "2026-04-11T02:32:46Z")
Problem Description When deploying the Qwen3.5-122B-A10B model on 8 Ascend 910B NPUs (not single card) via the Ascend-adapted version of vLLM, testing model accuracy with a 1,319-sample gsm8k\_gen dataset results in criti… --- ## [Any project supported plan for minicpm-o-4.5？](https://discuss.vllm.ai/t/any-project-supported-plan-for-minicpm-o-4-5/2492)
**Author:** [@pzc163](https://discuss.vllm.ai/u/pzc163)\ **Replies:** 1\ **Last updated:** [March 26, 2026, 7:50am UTC](https://discuss.vllm.ai/t/any-project-supported-plan-for-minicpm-o-4-5/2492 "2026-03-26T07:50:18Z")
I would like to ask if the vllm-omni team has plans to support minicpm-o-4.5. As the first full-duplex open-source multimodal model, I haven’t seen a roadmap from vllm-omni supporting minicpm-o-4.5 yet. --- ## [Trying to run Qwen3.5-397B-A17B-GPTQ-Int4](https://discuss.vllm.ai/t/trying-to-run-qwen3-5-397b-a17b-gptq-int4/2450)
**Author:** [@sinewave-2097](https://discuss.vllm.ai/u/sinewave-2097)\ **Replies:** 10\ **Last updated:** [March 13, 2026, 4:52am UTC](https://discuss.vllm.ai/t/trying-to-run-qwen3-5-397b-a17b-gptq-int4/2450 "2026-03-13T04:52:55Z")
@RunLLM , I’m trying to run Qwen3.5-397B-A17B-GPTQ-Int4 on Ubuntu 22. Is this supported? My error message: vllm-qwen35-gptq-int4 | (APIServer pid=1) INFO 03-12 23:12:44 \[utils.py:293\] vllm-qwen35-gptq-int4 | (APIServ… --- ## [Suggestion to improve inferencing speed](https://discuss.vllm.ai/t/suggestion-to-improve-inferencing-speed/2080)
**Author:** [@mridulrao](https://discuss.vllm.ai/u/mridulrao)\ **Replies:** 17\ **Last updated:** [March 11, 2026, 7:19am UTC](https://discuss.vllm.ai/t/suggestion-to-improve-inferencing-speed/2080 "2026-03-11T07:19:15Z")
I am running Qwen/Qwen3-4B-Instruct-2507 on vLLM 0.6.x using an NVIDIA L4 (24GB) GPU, and I am trying to optimize latency for a real-time voice application. My deployment uses the following vLLM configuration: python3 … --- ## [Critique my vLLM configuration for qwen3-coder-next](https://discuss.vllm.ai/t/critique-my-vllm-configuration-for-qwen3-coder-next/2434)
**Author:** [@sinewave-2097](https://discuss.vllm.ai/u/sinewave-2097)\ **Replies:** 3\ **Last updated:** [March 10, 2026, 6:42pm UTC](https://discuss.vllm.ai/t/critique-my-vllm-configuration-for-qwen3-coder-next/2434 "2026-03-10T18:42:21Z")
@RunLLM I’m running vLLM 0.17.0 in a docker container on Ubuntu 22 and serving qwen3-coder-next with 8 qty A6000 GPUs. Please review my docker config and suggest improvements. services: vllm: image: vllm/vllm-open… --- ## [Hosting Qwen 3.5 35B-A3B model](https://discuss.vllm.ai/t/hosting-qwen-3-5-35b-a3b-model/2403)
**Author:** [@san7988](https://discuss.vllm.ai/u/san7988)\ **Replies:** 1\ **Last updated:** [February 25, 2026, 7:46pm UTC](https://discuss.vllm.ai/t/hosting-qwen-3-5-35b-a3b-model/2403 "2026-02-25T19:46:44Z")
I’m trying to start vllm server with Qwen 3.5 35B-A3B model. However, the startup fails. Below is my command and error: CUDA\_VISIBLE\_DEVICES=6 VLLM\_USE\_TRITON\_MOE=0 vllm serve Qwen/Qwen3.5-35B-A3B --tensor-parallel-si… --- ## [Mistral Small 3.2 finetune errors out: There is no module or parameter named 'language\_model' in LlamaForCausalLM](https://discuss.vllm.ai/t/mistral-small-3-2-finetune-errors-out-there-is-no-module-or-parameter-named-language-model-in-llamaforcausallm/1764)
**Author:** [@mags0ft](https://discuss.vllm.ai/u/mags0ft)\ **Replies:** 3\ **Last updated:** [February 18, 2026, 5:39pm UTC](https://discuss.vllm.ai/t/mistral-small-3-2-finetune-errors-out-there-is-no-module-or-parameter-named-language-model-in-llamaforcausallm/1764 "2026-02-18T17:39:32Z")
Hello there! I worked on a fine-tune of Mistral Small 3.2 2506 Instruct for some time now, and I just cannot seem to get it working with vLLM. The standard HF repo mistralai/Mistral-Small-3.2-24B-Instruct-2506 runs perf… --- ## [Mistral-small-3.2: Unable to locate consolidated.safetensors.index.json](https://discuss.vllm.ai/t/mistral-small-3-2-unable-to-locate-consolidated-safetensors-index-json/2353)
**Author:** [@xjyk](https://discuss.vllm.ai/u/xjyk)\ **Replies:** 1\ **Last updated:** [February 8, 2026, 7:05am UTC](https://discuss.vllm.ai/t/mistral-small-3-2-unable-to-locate-consolidated-safetensors-index-json/2353 "2026-02-08T07:05:17Z")
When I’m attempting to serve \`mistral-small-3.2\`, vllm is unable to locate consolidated.safetensors.index.json Error: (EngineCore\_DP0 pid=63) INFO 02-08 06:51:32 \[weight\_utils.py:567\] No consolidated.safetensors.index.j… --- ## [How to run Deep Seek OCR 2 in vllm](https://discuss.vllm.ai/t/how-to-run-deep-seek-ocr-2-in-vllm/2280)
**Author:** [@mahimairaja](https://discuss.vllm.ai/u/mahimairaja)\ **Replies:** 1\ **Last updated:** [January 27, 2026, 2:49pm UTC](https://discuss.vllm.ai/t/how-to-run-deep-seek-ocr-2-in-vllm/2280 "2026-01-27T14:49:26Z")
I tried to run the deepseek ocr model in Colab with T4 GPU from vllm import LLM, SamplingParams from vllm.model\_executor.models.deepseek\_ocr import NGramPerReqLogitsProcessor from PIL import Image # Create model insta… --- ## [Vllm-omni cannot load z-image-turbo](https://discuss.vllm.ai/t/vllm-omni-cannot-load-z-image-turbo/2158)
**Author:** [@wkf2000](https://discuss.vllm.ai/u/wkf2000)\ **Replies:** 3\ **Last updated:** [December 27, 2025, 5:00pm UTC](https://discuss.vllm.ai/t/vllm-omni-cannot-load-z-image-turbo/2158 "2025-12-27T17:00:53Z")
docker image: vllm-omni:v0.11.0rc1 model: Tongyi-MAI/Z-Image-Turbo command: vllm serve /model/Z-Image-Turbo --served-model-name z-image-turbo --port 18005 --host 0.0.0.0 --tensor-parallel-size 1 --max-model-len 32768 … --- ## [Llama 3.3 70B very slow](https://discuss.vllm.ai/t/llama-3-3-70b-very-slow/1715)
**Author:** [@DavidNemeskey](https://discuss.vllm.ai/u/DavidNemeskey)\ **Replies:** 5\ **Last updated:** [December 11, 2025, 3:45pm UTC](https://discuss.vllm.ai/t/llama-3-3-70b-very-slow/1715 "2025-12-11T15:45:43Z")
Hi, I have been testing Llama 3.3 70B on 2 H200 GPUs, and I found it to be very slow. I can get around 40 tokens per second, compared to gpt-oss-120B (200 tokens) or GLM-4.5-Air FP8 (120 tokens). I am wondering if this… --- ## [Text to speech support with /v1/audio/speech route](https://discuss.vllm.ai/t/text-to-speech-support-with-v1-audio-speech-route/2046)
**Author:** [@dotmobo](https://discuss.vllm.ai/u/dotmobo)\ **Replies:** 1\ **Last updated:** [November 28, 2025, 8:47am UTC](https://discuss.vllm.ai/t/text-to-speech-support-with-v1-audio-speech-route/2046 "2025-11-28T08:47:59Z")
Hello everyone! I’m using vLLM for chat models, reranking, embedding, and speech-to-text. The only use case missing in vLLM is support for text-to-speech models like Piper and Kokoro. Could we consider adding support for… --- ## [Using InternVL3 to perform OCR tasks yields worse results in vLLM than in LMDeploy](https://discuss.vllm.ai/t/using-internvl3-to-perform-ocr-tasks-yields-worse-results-in-vllm-than-in-lmdeploy/2042)
**Author:** [@jianworks](https://discuss.vllm.ai/u/jianworks)\ **Replies:** 2\ **Last updated:** [November 27, 2025, 6:18am UTC](https://discuss.vllm.ai/t/using-internvl3-to-perform-ocr-tasks-yields-worse-results-in-vllm-than-in-lmdeploy/2042 "2025-11-27T06:18:17Z")
Has anyone used InternVL3 to perform OCR tasks on vLLM? Our tests show that the same InternVL3 model performs significantly worse on vLLM than on LMDeploy. Both are executed as command lines with parameters as consistent… --- ## [Serving minimax-m2](https://discuss.vllm.ai/t/serving-minimax-m2/1870)
**Author:** [@t.white](https://discuss.vllm.ai/u/t.white)\ **Replies:** 3\ **Last updated:** [November 8, 2025, 5:21am UTC](https://discuss.vllm.ai/t/serving-minimax-m2/1870 "2025-11-08T05:21:02Z")
I downloaded minimax-m2 from huggingface to run offline. I followed the minimax m2 recipe and cannot get it to work. I ran uv venv source .venv/bin/activate uv pip install 'triton-kernels @ git+https://github.com/trit… --- ## [Disabling reasoning of Qwen3-VL-8B-Thinking per request](https://discuss.vllm.ai/t/disabling-reasoning-of-qwen3-vl-8b-thinking-per-request/1800)
**Author:** [@milos](https://discuss.vllm.ai/u/milos)\ **Replies:** 1\ **Last updated:** [October 29, 2025, 10:23pm UTC](https://discuss.vllm.ai/t/disabling-reasoning-of-qwen3-vl-8b-thinking-per-request/1800 "2025-10-29T22:23:44Z")
Hello, I started my vLLM instance with Qwen3-VL-8B-Thinking model with this script: sudo docker run -d --runtime=nvidia --gpus "device=0" \\ -p 8002:8000 --name q8t \\ -v ~/.cache/huggingface:/root/.cache/huggingface… --- ## [Zerank - deploying using vllm](https://discuss.vllm.ai/t/zerank-deploying-using-vllm/1793)
**Author:** [@WarriorOfLiberation](https://discuss.vllm.ai/u/WarriorOfLiberation)\ **Replies:** 3\ **Last updated:** [October 29, 2025, 8:29am UTC](https://discuss.vllm.ai/t/zerank-deploying-using-vllm/1793 "2025-10-29T08:29:02Z")
How to host zeroentropy’s zerank-small using vllm? --- ## [Vllm-ascend是否支持deepseek-ocr](https://discuss.vllm.ai/t/vllm-ascend-deepseek-ocr/1754)
**Author:** [@whwususu](https://discuss.vllm.ai/u/whwususu)\ **Replies:** 2\ **Last updated:** [October 21, 2025, 3:22pm UTC](https://discuss.vllm.ai/t/vllm-ascend-deepseek-ocr/1754 "2025-10-21T15:22:24Z")
vllm-ascend是否支持deepseek-ocr部署？ --- ## [Custom edit on the embedding out](https://discuss.vllm.ai/t/custom-edit-on-the-embedding-out/1725)
**Author:** [@wj210](https://discuss.vllm.ai/u/wj210)\ **Replies:** 2\ **Last updated:** [October 15, 2025, 4:34am UTC](https://discuss.vllm.ai/t/custom-edit-on-the-embedding-out/1725 "2025-10-15T04:34:51Z")
Is it possible to perform an additional matrix mulitplication on the embedding outputs (post-embedding layer) on token positions that are dynamic across samples? Furthermore, this multiplication must be performed on the… [Next page](https://discuss.vllm.ai/c/model-support/10.md?page=1)