# General **URL:** https://discuss.vllm.ai/c/general/4.md [Latest](https://discuss.vllm.ai/latest.md) · [Categories](https://discuss.vllm.ai/categories.md) --- ## [Welcome to vLLM Forums! 👋](https://discuss.vllm.ai/t/welcome-to-vllm-forums/5)
**Author:** [@system](https://discuss.vllm.ai/u/system)\ **Replies:** 1\ **Last updated:** [March 24, 2025, 12:10am UTC](https://discuss.vllm.ai/t/welcome-to-vllm-forums/5 "2025-03-24T00:10:17Z")
We are so glad you joined us. vLLM Forums A high-throughput and memory-efficient inference and serving engine for LLMs Here are some things you can do to get started: :speaking\_head: Introduce yourself by adding you… --- ## [About the General category](https://discuss.vllm.ai/t/about-the-general-category/3)
**Author:** [@system](https://discuss.vllm.ai/u/system)\ **Replies:** 0
--- ## [Some requests never complete: token generation keeps running but the response never finishes (looks like an infinite generation loop)](https://discuss.vllm.ai/t/some-requests-never-complete-token-generation-keeps-running-but-the-response-never-finishes-looks-like-an-infinite-generation-loop/3080)
**Author:** [@Coder5424](https://discuss.vllm.ai/u/Coder5424)\ **Replies:** 1\ **Last updated:** [October 22, 2026, 4:18pm UTC](https://discuss.vllm.ai/t/some-requests-never-complete-token-generation-keeps-running-but-the-response-never-finishes-looks-like-an-infinite-generation-loop/3080 "2026-10-22T16:18:41Z")
I’m new to vLLM and self-hosting LLMs in general, so please bear with me if I’ve missed anything obvious. I’m serving QuantTrio/Qwen3.5-35B-A3B-AWQ using vllm/vllm-openai:v0.31.0 on two RTX 4090 GPUs with 24 GB VRAM eac… --- ## [vLLM 0.20.1, Radeon AI 9700, 1 CPU core at 100%](https://discuss.vllm.ai/t/vllm-0-20-1-radeon-ai-9700-1-cpu-core-at-100/2621)
**Author:** [@dramallama](https://discuss.vllm.ai/u/dramallama)\ **Replies:** 8\ **Last updated:** [October 21, 2026, 11:38am UTC](https://discuss.vllm.ai/t/vllm-0-20-1-radeon-ai-9700-1-cpu-core-at-100/2621 "2026-10-21T11:38:05Z")
Hello there, I’m serving a FP8-quant of gemma-4 E4B on a Radeon AI Pro 9700 with vLLM and I find that things perform and scale wonderfully, so thanks a lot to the vLLM team! However, even when no requests are processed… --- ## [/v1/systemone with Cloudflare/clef-flash not working on v0.32.0](https://discuss.vllm.ai/t/v1-systemone-with-cloudflare-clef-flash-not-working-on-v0-32-0/3067)
**Author:** [@jgr](https://discuss.vllm.ai/u/jgr)\ **Replies:** 1\ **Last updated:** [October 19, 2026, 2:18pm UTC](https://discuss.vllm.ai/t/v1-systemone-with-cloudflare-clef-flash-not-working-on-v0-32-0/3067 "2026-10-19T14:18:55Z")
Hello I’m trying to run Cloudflare/clef-flash locally, with the following setup: vllm serve /models/clef-flash –served-model-name Cloudflare/clef-flash –host 0.0.0.0 –port 5045 –max-model-len 16384 –gpu-memory-uti… --- ## [Question about max\_num\_seqs](https://discuss.vllm.ai/t/question-about-max-num-seqs/3042)
**Author:** [@krn](https://discuss.vllm.ai/u/krn)\ **Replies:** 1\ **Last updated:** [October 9, 2026, 7:34pm UTC](https://discuss.vllm.ai/t/question-about-max-num-seqs/3042 "2026-10-09T19:34:54Z")
Hi all, We’re running Qwen3.8-27B on vLLM for 700+ users, and our benchmarks showed a performance knee at around 128 concurrent requests with the default settings. Since vLLM already handles scheduling and queuing dyna… --- ## [Question about --enable-chunked-prefill and --max-num-batched-tokens](https://discuss.vllm.ai/t/question-about-enable-chunked-prefill-and-max-num-batched-tokens/3039)
**Author:** [@Pigcanflysohigh](https://discuss.vllm.ai/u/Pigcanflysohigh)\ **Replies:** 2\ **Last updated:** [October 8, 2026, 10:14am UTC](https://discuss.vllm.ai/t/question-about-enable-chunked-prefill-and-max-num-batched-tokens/3039 "2026-10-08T10:14:03Z")
What is the functions of --enable-chunked-prefill and --max-num-batched-tokens? make inferen faster or something else？ --- ## [Handling request timeouts and KV cache bloat during bursty async workloads in vLLM](https://discuss.vllm.ai/t/handling-request-timeouts-and-kv-cache-bloat-during-bursty-async-workloads-in-vllm/3034)
**Author:** [@iope](https://discuss.vllm.ai/u/iope)\ **Replies:** 1\ **Last updated:** [October 7, 2026, 12:03pm UTC](https://discuss.vllm.ai/t/handling-request-timeouts-and-kv-cache-bloat-during-bursty-async-workloads-in-vllm/3034 "2026-10-07T12:03:43Z")
We are currently integrating a self-hosted vLLM deployment on an A100 80GB GPU to handle automated text summarization for our automated dachecker pipeline. The workflow processes batches of unstructured web text and doma… --- ## [Audio\_tower module missing](https://discuss.vllm.ai/t/audio-tower-module-missing/3027)
**Author:** [@AnttiK](https://discuss.vllm.ai/u/AnttiK)\ **Replies:** 1\ **Last updated:** [October 5, 2026, 4:51am UTC](https://discuss.vllm.ai/t/audio-tower-module-missing/3027 "2026-10-05T04:51:25Z")
ValueError: There is no module or parameter named ‘audio\_tower.layers.0.self\_attn.k\_proj.linear.weight’ in Gemma4ForConditionalGeneration. with vllm version 0.31.0 and 0.30.1rc1.dev630+g18f8f9602 vllm/vllm-openai-rocm:… --- ## [Is provider KV caching sufficient for agent swarms and long run agents?](https://discuss.vllm.ai/t/is-provider-kv-caching-sufficient-for-agent-swarms-and-long-run-agents/3006)
**Author:** [@AgilePenguin36](https://discuss.vllm.ai/u/AgilePenguin36)\ **Replies:** 2\ **Last updated:** [October 1, 2026, 10:56pm UTC](https://discuss.vllm.ai/t/is-provider-kv-caching-sufficient-for-agent-swarms-and-long-run-agents/3006 "2026-10-01T22:56:04Z")
I’m trying to build a side project in the inference space. I’ve been talking to a few inference engineers and startups and I’ve been hearing how annoying it is to not have manual control over the KV cache at times and ju… --- ## [Methods for Server-Side Parameter Optimization in VLLM Performance Benchmarking](https://discuss.vllm.ai/t/methods-for-server-side-parameter-optimization-in-vllm-performance-benchmarking/3012)
**Author:** [@xinbaby-1](https://discuss.vllm.ai/u/xinbaby-1)\ **Replies:** 1\ **Last updated:** [September 29, 2026, 9:04am UTC](https://discuss.vllm.ai/t/methods-for-server-side-parameter-optimization-in-vllm-performance-benchmarking/3012 "2026-09-29T09:04:38Z")
1 - Overview Since system performance is affected by the available GPU memory for inference, and the available memory for inference is in turn constrained by the server-side parameters of the VLLM framework, choosing ap… --- ## [Presentation on context, expert, data and tensor parallelism in vLLM](https://discuss.vllm.ai/t/presentation-on-context-expert-data-and-tensor-parallelism-in-vllm/2994)
**Author:** [@ankur6ue](https://discuss.vllm.ai/u/ankur6ue)\ **Replies:** 1\ **Last updated:** [September 22, 2026, 12:17pm UTC](https://discuss.vllm.ai/t/presentation-on-context-expert-data-and-tensor-parallelism-in-vllm/2994 "2026-09-22T12:17:44Z")
Hello everyone, I’ve been on a rabbit hole of how Data, Tensor, Context and Expert parallelism come together in vLLM.. I have created a presentation here: It covers: Multi-headed attention and tensor parallelism Dat… --- ## [Add LLM observability data management](https://discuss.vllm.ai/t/add-llm-observability-data-management/2978)
**Author:** [@zeding2019](https://discuss.vllm.ai/u/zeding2019)\ **Replies:** 2\ **Last updated:** [September 16, 2026, 3:19am UTC](https://discuss.vllm.ai/t/add-llm-observability-data-management/2978 "2026-09-16T03:19:22Z")
I want to add a system of LLM observability data management, a component similar to Alibaba’s Druid. It exports runtime metrics, including request-lifecycle metrics such as block utilization, KV cache space, and read/wri… --- ## [Vllm推理指标如何做可视化？](https://discuss.vllm.ai/t/vllm/2179)
**Author:** [@sxcnmslll](https://discuss.vllm.ai/u/sxcnmslll)\ **Replies:** 2\ **Last updated:** [September 16, 2026, 3:11am UTC](https://discuss.vllm.ai/t/vllm/2179 "2026-09-16T03:11:03Z")
vllm有什么自带工具，或者其他项目的工具，可以将推理中的各项开销（如显存分配，prefill时长，decode时长等）做可视化吗 --- ## [\[RFC & PR Stack\] Application-Directed Prefix Checkpoints for Hybrid GDN/Mamba (Up to 7.6x Speedup on L40S)](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949)
**Author:** [@nicholaskh-ai](https://discuss.vllm.ai/u/nicholaskh-ai)\ **Replies:** 6\ **Last updated:** [September 15, 2026, 12:59pm UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949 "2026-09-15T12:59:02Z")
Hi vLLM Community & Maintainers, We have proposed an RFC and a 3-part PR stack introducing Application-Directed Prefix Checkpointing and Batched Grouped Prefill Execution for hybrid GDN/Mamba architectures (such as Qwen… --- ## [vLLM-0.23.0 lora加载问题](https://discuss.vllm.ai/t/vllm-0-23-0-lora/2976)
**Author:** [@ymz12das](https://discuss.vllm.ai/u/ymz12das)\ **Replies:** 1\ **Last updated:** [September 15, 2026, 6:12am UTC](https://discuss.vllm.ai/t/vllm-0-23-0-lora/2976 "2026-09-15T06:12:16Z")
对Qwen3.8-27B进行lora训练后，lora对模块为q\_proj/k\_proj/v\_proj/o\_proj），对in\_proj\_a 、in\_proj\_b 、in\_proj\_z 、in\_proj\_qkv 、out\_proj 后续使用vLLM加载base模型和lora适配器时，模型一直输出！ 但是将lora适配器与模型合并后再用vLLM加载时输出正常 这是什么原因？ --- ## [vLLM 0.29 7900xtx or 890m 1 cpu core 100%](https://discuss.vllm.ai/t/vllm-0-29-7900xtx-or-890m-1-cpu-core-100/2975)
**Author:** [@AnttiK](https://discuss.vllm.ai/u/AnttiK)\ **Replies:** 8\ **Last updated:** [September 14, 2026, 5:15pm UTC](https://discuss.vllm.ai/t/vllm-0-29-7900xtx-or-890m-1-cpu-core-100/2975 "2026-09-14T17:15:44Z")
Hi, Multiple AMD cards (7900xtx and 890m at least what I have) are again having the same old problem, 1 cpu core per gpu are having 100% utilization even when idle. 100,0% VLLM::EngineCor I have the -e VLLM\_SLEEP\_WHEN… --- ## [Muse glimmer - how to disable thinking](https://discuss.vllm.ai/t/muse-glimmer-how-to-disable-thinking/2970)
**Author:** [@AnttiK](https://discuss.vllm.ai/u/AnttiK)\ **Replies:** 1\ **Last updated:** [September 11, 2026, 9:57pm UTC](https://discuss.vllm.ai/t/muse-glimmer-how-to-disable-thinking/2970 "2026-09-11T21:57:18Z")
how to disable muse glimmer thinking? on vllm --- ## [Not able to run amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 on rocm](https://discuss.vllm.ai/t/not-able-to-run-amd-qwen3-8-27b-quark-awq-int4-w4a16-on-rocm/2954)
**Author:** [@AnttiK](https://discuss.vllm.ai/u/AnttiK)\ **Replies:** 1\ **Last updated:** [September 9, 2026, 9:56pm UTC](https://discuss.vllm.ai/t/not-able-to-run-amd-qwen3-8-27b-quark-awq-int4-w4a16-on-rocm/2954 "2026-09-09T21:56:58Z")
how to serve: amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 on docker vllm rocm latest? it throws me errors like AttributeError: ‘dict’ object has no attribute ‘endswith’ \[rank0\]:\[W909 21:53:54.659294217 ProcessGroupNCCL.cpp… --- ## [vLLM-Ascend在银河麒麟桌面版上不开Eager就报错，开了性能暴跌，换服务器版就正常，这是系统差异导致的吗？](https://discuss.vllm.ai/t/vllm-ascend-eager/2938)
**Author:** [@oeyoews](https://discuss.vllm.ai/u/oeyoews)\ **Replies:** 1\ **Last updated:** [September 4, 2026, 6:08am UTC](https://discuss.vllm.ai/t/vllm-ascend-eager/2938 "2026-09-04T06:08:13Z")
vllm-ascend（310p) 是否支持银河麒麟桌面版操作系统， 如果不开启eager mode, 就会提示下面这个报错， 但是如果开启eager mode, 吞吐 仅仅只有 5~6 tokens/s ， 如果把 桌面版操作系统换成银河麒麟服务器版本， 不用开启eager的情况下，吞吐速度就会有成倍的提升 ant to get the accurate stacktrace, please set the environment… --- ## [我用单卡跑通了deepseek-v4-flash-0731的dspark模式](https://discuss.vllm.ai/t/deepseek-v4-flash-0731-dspark/2914)
**Author:** [@ChongqingBigShark](https://discuss.vllm.ai/u/ChongqingBigShark)\ **Replies:** 1\ **Last updated:** [August 22, 2026, 11:35am UTC](https://discuss.vllm.ai/t/deepseek-v4-flash-0731-dspark/2914 "2026-08-22T11:35:14Z")
(APIServer pid=17) INFO 08-22 18:39:39 \[api\_utils.py:345\] (APIServer pid=17) INFO 08-22 18:39:39 \[api\_utils.py:345\] █ █ █▄ ▄█ (APIServer pid=17) INFO 08-22 18:39:39 \[api\_utils.py:345\] ▄▄ ▄█ █ █ … --- ## [Flash-0731 在 vLLM 0.27.1 + SM120 上 dspark 不可启动](https://discuss.vllm.ai/t/flash-0731-vllm-0-27-1-sm120-dspark/2912)
**Author:** [@ChongqingBigShark](https://discuss.vllm.ai/u/ChongqingBigShark)\ **Replies:** 1\ **Last updated:** [August 21, 2026, 2:05pm UTC](https://discuss.vllm.ai/t/flash-0731-vllm-0-27-1-sm120-dspark/2912 "2026-08-21T14:05:30Z")
vLLM 0.27.1 源码 bug——主模型和 DSpark 草稿模型共用同一个 marlin 专家类，但主模型的 named\_parameters() 里有 w13\_weight\_scale，草稿模型的却没有（参数注册不一致）。默认 DeepGEMM 下不暴露，切到 marlin 才触发。参数层面无解。——MXFP4 专家四个后端全灭： DeepGEMM：布局断言崩 Triton：源码排除 SM120 flashinfer\_cu… --- ## [Why does changing VLLM\_CACHE\_ROOT not reproduce cold-start latency?](https://discuss.vllm.ai/t/why-does-changing-vllm-cache-root-not-reproduce-cold-start-latency/2910)
**Author:** [@harry](https://discuss.vllm.ai/u/harry)\ **Replies:** 1\ **Last updated:** [August 21, 2026, 2:05pm UTC](https://discuss.vllm.ai/t/why-does-changing-vllm-cache-root-not-reproduce-cold-start-latency/2910 "2026-08-21T14:05:19Z")
I’m running a vLLM server inside a container and using VLLM\_CACHE\_ROOT to specify the cache directory. On the container’s first startup, after launching the vLLM server for the first time, the latency of the first reque… --- ## [Model path in compute\_hash() prevents torch.compile cache reuse across identical model loads](https://discuss.vllm.ai/t/model-path-in-compute-hash-prevents-torch-compile-cache-reuse-across-identical-model-loads/2908)
**Author:** [@twb1235](https://discuss.vllm.ai/u/twb1235)\ **Replies:** 1\ **Last updated:** [August 21, 2026, 2:04pm UTC](https://discuss.vllm.ai/t/model-path-in-compute-hash-prevents-torch-compile-cache-reuse-across-identical-model-loads/2908 "2026-08-21T14:04:38Z")
The model field (HuggingFace model name or local path) is currently included in ModelConfig.compute\_hash(), which means two instances serving the exact same model from different paths will compute different config\_hash v… --- ## [Mooncake prefix information in storage](https://discuss.vllm.ai/t/mooncake-prefix-information-in-storage/2904)
**Author:** [@svg](https://discuss.vllm.ai/u/svg)\ **Replies:** 3\ **Last updated:** [August 21, 2026, 5:08am UTC](https://discuss.vllm.ai/t/mooncake-prefix-information-in-storage/2904 "2026-08-21T05:08:43Z")
Is it true that In MoonCake, the prefix information disappears when it reaches the storage? Can the prefix information be preserved for use by the eviction algorithm? --- ## [Vllm是否支持加载deepseek\_ocr模型的lora适配器进行推理](https://discuss.vllm.ai/t/vllm-deepseek-ocr-lora/2161)
**Author:** [@sanjer](https://discuss.vllm.ai/u/sanjer)\ **Replies:** 3\ **Last updated:** [August 21, 2026, 12:14am UTC](https://discuss.vllm.ai/t/vllm-deepseek-ocr-lora/2161 "2026-08-21T00:14:57Z")
使用ms-swift对deepseek\_ocr模型进行lora微调得到lora适配器（正常的，lora未合并），但接下来使用vllm最新版本v0.13.0来加载这个适配器推理时报错了 self.model = self.load\_lora\_model( (EngineCore\_DP0 pid=10754) ERROR 12-28 14:53:45 \[core.py:866\] ^^^^^^^^^^^^… --- ## [Dlash2 無法使用 在 vllm 0.27.1](https://discuss.vllm.ai/t/dlash2-vllm-0-27-1/2894)
**Author:** [@kuolung1](https://discuss.vllm.ai/u/kuolung1)\ **Replies:** 1\ **Last updated:** [August 19, 2026, 8:48am UTC](https://discuss.vllm.ai/t/dlash2-vllm-0-27-1/2894 "2026-08-19T08:48:10Z")
vllm serve Qwen/Qwen3.8-27B –speculative-config ‘{ “method”: “dflash”, “model”: “incoai/Qwen3.8-27B-DFlash2”, “num\_speculative\_tokens”: 7 }’ vllm 0.27.1 (APIServer pid=1116141) File “/home/user/vLLM/env0271/lib/… --- ## [Expert Parallelism All-to-All Communication without NVLink and DeepEP](https://discuss.vllm.ai/t/expert-parallelism-all-to-all-communication-without-nvlink-and-deepep/2421)
**Author:** [@Tianchen2001](https://discuss.vllm.ai/u/Tianchen2001)\ **Replies:** 4\ **Last updated:** [August 17, 2026, 9:12pm UTC](https://discuss.vllm.ai/t/expert-parallelism-all-to-all-communication-without-nvlink-and-deepep/2421 "2026-08-17T21:12:04Z")
Hi, I would like to deploy Deepseek-V2-Lite on my computer with 2 NVIDIA 3090 GPUs. There is not P2P communication channel between GPUs. By default, it seems to activate all-reduce-scatter for MoE layer. However, is ther… --- ## [FLASHINFER is FASTER then FA2 on Ampere hardware](https://discuss.vllm.ai/t/flashinfer-is-faster-then-fa2-on-ampere-hardware/2883)
**Author:** [@Frenzy\_Biscuit](https://discuss.vllm.ai/u/Frenzy_Biscuit)\ **Replies:** 5\ **Last updated:** [August 16, 2026, 7:32pm UTC](https://discuss.vllm.ai/t/flashinfer-is-faster-then-fa2-on-ampere-hardware/2883 "2026-08-16T19:32:35Z")
Hardware: 4x3090 To recap on my issues with FA2: MTP mode on either gemma 4 31B IT or Qwen 3.8 27B was slow, especially at longer contexts; Disabling MTP resolved the slowness, but was still slower then expected on ra… --- ## [Support for Qwen3.6 (27B & 35B-A3B) Hybrid Gated DeltaNet Models on TPU v5e-8 with vLLM and SGLang](https://discuss.vllm.ai/t/support-for-qwen3-6-27b-35b-a3b-hybrid-gated-deltanet-models-on-tpu-v5e-8-with-vllm-and-sglang/2875)
**Author:** [@arjuntech-07](https://discuss.vllm.ai/u/arjuntech-07)\ **Replies:** 12\ **Last updated:** [October 16, 2026, 7:09pm UTC](https://discuss.vllm.ai/t/support-for-qwen3-6-27b-35b-a3b-hybrid-gated-deltanet-models-on-tpu-v5e-8-with-vllm-and-sglang/2875 "2026-10-16T19:09:38Z")
Hi vLLM Community, I am looking to deploy the new hybrid linear-attention models from Alibaba—specifically Qwen/Qwen3.6-27B (dense) and Qwen/Qwen3.6-35B-A3B (MoE)—on a Google Cloud TPU v5e-8 instance (8 chips, 128 GB to… [Next page](https://discuss.vllm.ai/c/general/4.md?page=1)