# Latest **URL:** https://discuss.vllm.ai/latest.md [Latest](https://discuss.vllm.ai/latest.md) · [Categories](https://discuss.vllm.ai/categories.md) --- ## [Welcome to vLLM Forums! 👋](https://discuss.vllm.ai/t/welcome-to-vllm-forums/5)
**Author:** [@system](https://discuss.vllm.ai/u/system)\ **Replies:** 1\ **Last updated:** [March 24, 2025, 12:10am UTC](https://discuss.vllm.ai/t/welcome-to-vllm-forums/5 "2025-03-24T00:10:17Z")
We are so glad you joined us. vLLM Forums A high-throughput and memory-efficient inference and serving engine for LLMs Here are some things you can do to get started: :speaking\_head: Introduce yourself by adding you… --- ## [Presentation on context, expert, data and tensor parallelism in vLLM](https://discuss.vllm.ai/t/presentation-on-context-expert-data-and-tensor-parallelism-in-vllm/2994)
**Author:** [@ankur6ue](https://discuss.vllm.ai/u/ankur6ue)\ **Replies:** 1\ **Last updated:** [September 22, 2026, 12:17pm UTC](https://discuss.vllm.ai/t/presentation-on-context-expert-data-and-tensor-parallelism-in-vllm/2994 "2026-09-22T12:17:44Z")
Hello everyone, I’ve been on a rabbit hole of how Data, Tensor, Context and Expert parallelism come together in vLLM.. I have created a presentation here: It covers: Multi-headed attention and tensor parallelism Dat… --- ## [Title: OutOfResources: shared memory (98304 \> 65536) on Turing (SM75) — global attention layers with head\_dim=512](https://discuss.vllm.ai/t/title-outofresources-shared-memory-98304-65536-on-turing-sm75-global-attention-layers-with-head-dim-512/2990)
**Author:** [@GentleHarbor25](https://discuss.vllm.ai/u/GentleHarbor25)\ **Replies:** 2\ **Last updated:** [September 20, 2026, 7:13am UTC](https://discuss.vllm.ai/t/title-outofresources-shared-memory-98304-65536-on-turing-sm75-global-attention-layers-with-head-dim-512/2990 "2026-09-20T07:13:02Z")
Title: OutOfResources: shared memory (98304 \> 65536) on Turing (SM75) — global attention layers with head\_dim=512 Summary I am unable to serve this checkpoint on NVIDIA Turing (compute capability 7.5) GPUs. vLLM fails d… --- ## [vLLM Optimizer — automated benchmark-driven configuration tuning for vLLM](https://discuss.vllm.ai/t/vllm-optimizer-automated-benchmark-driven-configuration-tuning-for-vllm/2985)
**Author:** [@Iaagl](https://discuss.vllm.ai/u/Iaagl)\ **Replies:** 1\ **Last updated:** [September 18, 2026, 8:37pm UTC](https://discuss.vllm.ai/t/vllm-optimizer-automated-benchmark-driven-configuration-tuning-for-vllm/2985 "2026-09-18T20:37:45Z")
Hi everyone, I’ve been working on an open-source project called vLLM Optimizer. The idea for the project came to me while I was optimizing a model and found myself repeating the same deployment and benchmarking steps o… --- ## [Add LLM observability data management](https://discuss.vllm.ai/t/add-llm-observability-data-management/2978)
**Author:** [@zeding2019](https://discuss.vllm.ai/u/zeding2019)\ **Replies:** 2\ **Last updated:** [September 16, 2026, 3:19am UTC](https://discuss.vllm.ai/t/add-llm-observability-data-management/2978 "2026-09-16T03:19:22Z")
I want to add a system of LLM observability data management, a component similar to Alibaba’s Druid. It exports runtime metrics, including request-lifecycle metrics such as block utilization, KV cache space, and read/wri… --- ## [Vllm推理指标如何做可视化？](https://discuss.vllm.ai/t/vllm/2179)
**Author:** [@sxcnmslll](https://discuss.vllm.ai/u/sxcnmslll)\ **Replies:** 2\ **Last updated:** [September 16, 2026, 3:11am UTC](https://discuss.vllm.ai/t/vllm/2179 "2026-09-16T03:11:03Z")
vllm有什么自带工具，或者其他项目的工具，可以将推理中的各项开销（如显存分配，prefill时长，decode时长等）做可视化吗 --- ## [\[RFC & PR Stack\] Application-Directed Prefix Checkpoints for Hybrid GDN/Mamba (Up to 7.6x Speedup on L40S)](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949)
**Author:** [@nicholaskh-ai](https://discuss.vllm.ai/u/nicholaskh-ai)\ **Replies:** 6\ **Last updated:** [September 15, 2026, 12:59pm UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949 "2026-09-15T12:59:02Z")
Hi vLLM Community & Maintainers, We have proposed an RFC and a 3-part PR stack introducing Application-Directed Prefix Checkpointing and Batched Grouped Prefill Execution for hybrid GDN/Mamba architectures (such as Qwen… --- ## [vLLM-0.23.0 lora加载问题](https://discuss.vllm.ai/t/vllm-0-23-0-lora/2976)
**Author:** [@ymz12das](https://discuss.vllm.ai/u/ymz12das)\ **Replies:** 1\ **Last updated:** [September 15, 2026, 6:12am UTC](https://discuss.vllm.ai/t/vllm-0-23-0-lora/2976 "2026-09-15T06:12:16Z")
对Qwen3.8-27B进行lora训练后，lora对模块为q\_proj/k\_proj/v\_proj/o\_proj），对in\_proj\_a 、in\_proj\_b 、in\_proj\_z 、in\_proj\_qkv 、out\_proj 后续使用vLLM加载base模型和lora适配器时，模型一直输出！ 但是将lora适配器与模型合并后再用vLLM加载时输出正常 这是什么原因？ --- ## [vLLM 0.29 7900xtx or 890m 1 cpu core 100%](https://discuss.vllm.ai/t/vllm-0-29-7900xtx-or-890m-1-cpu-core-100/2975)
**Author:** [@AnttiK](https://discuss.vllm.ai/u/AnttiK)\ **Replies:** 8\ **Last updated:** [September 14, 2026, 5:15pm UTC](https://discuss.vllm.ai/t/vllm-0-29-7900xtx-or-890m-1-cpu-core-100/2975 "2026-09-14T17:15:44Z")
Hi, Multiple AMD cards (7900xtx and 890m at least what I have) are again having the same old problem, 1 cpu core per gpu are having 100% utilization even when idle. 100,0% VLLM::EngineCor I have the -e VLLM\_SLEEP\_WHEN… --- ## [vLLM 0.20.1, Radeon AI 9700, 1 CPU core at 100%](https://discuss.vllm.ai/t/vllm-0-20-1-radeon-ai-9700-1-cpu-core-at-100/2621)
**Author:** [@dramallama](https://discuss.vllm.ai/u/dramallama)\ **Replies:** 6\ **Last updated:** [September 13, 2026, 12:23am UTC](https://discuss.vllm.ai/t/vllm-0-20-1-radeon-ai-9700-1-cpu-core-at-100/2621 "2026-09-13T00:23:29Z")
Hello there, I’m serving a FP8-quant of gemma-4 E4B on a Radeon AI Pro 9700 with vLLM and I find that things perform and scale wonderfully, so thanks a lot to the vLLM team! However, even when no requests are processed… --- ## [Muse glimmer - how to disable thinking](https://discuss.vllm.ai/t/muse-glimmer-how-to-disable-thinking/2970)
**Author:** [@AnttiK](https://discuss.vllm.ai/u/AnttiK)\ **Replies:** 1\ **Last updated:** [September 11, 2026, 9:57pm UTC](https://discuss.vllm.ai/t/muse-glimmer-how-to-disable-thinking/2970 "2026-09-11T21:57:18Z")
how to disable muse glimmer thinking? on vllm --- ## [Does setting TRITON\_CACHE\_DIR break the bundled (AOT) static-autotuner cubin restore?](https://discuss.vllm.ai/t/does-setting-triton-cache-dir-break-the-bundled-aot-static-autotuner-cubin-restore/2965)
**Author:** [@kaiste99](https://discuss.vllm.ai/u/kaiste99)\ **Replies:** 1\ **Last updated:** [September 11, 2026, 5:04pm UTC](https://discuss.vllm.ai/t/does-setting-triton-cache-dir-break-the-bundled-aot-static-autotuner-cubin-restore/2965 "2026-09-11T17:04:15Z")
Context vLLM 0.29.0 (pip install, not Docker), torch 2.13.0 NVIDIA GB10 (DGX Spark), single device, driver 580.173.02 / CUDA 13.0 Qwen/Qwen3.6-35B-A3B-FP8, --max-cudagraph-capture-size 8192, --max-model-len 262144, --… --- ## [Decode KV writeback: can multi-turn chat preserve cache reuse with default templates?](https://discuss.vllm.ai/t/decode-kv-writeback-can-multi-turn-chat-preserve-cache-reuse-with-default-templates/2955)
**Author:** [@lhecust](https://discuss.vllm.ai/u/lhecust)\ **Replies:** 1\ **Last updated:** [September 10, 2026, 1:50am UTC](https://discuss.vllm.ai/t/decode-kv-writeback-can-multi-turn-chat-preserve-cache-reuse-with-default-templates/2955 "2026-09-10T01:50:44Z")
Hi vLLM community, I would like to understand the practical benefits of Decode-phase KV writeback when server-side chat templates change the token sequence between conversation turns. This relates to the Decode-Phase K… --- ## [Not able to run amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 on rocm](https://discuss.vllm.ai/t/not-able-to-run-amd-qwen3-8-27b-quark-awq-int4-w4a16-on-rocm/2954)
**Author:** [@AnttiK](https://discuss.vllm.ai/u/AnttiK)\ **Replies:** 1\ **Last updated:** [September 9, 2026, 9:56pm UTC](https://discuss.vllm.ai/t/not-able-to-run-amd-qwen3-8-27b-quark-awq-int4-w4a16-on-rocm/2954 "2026-09-09T21:56:58Z")
how to serve: amd/Qwen3.8-27B-Quark-AWQ-INT4-W4A16 on docker vllm rocm latest? it throws me errors like AttributeError: ‘dict’ object has no attribute ‘endswith’ \[rank0\]:\[W909 21:53:54.659294217 ProcessGroupNCCL.cpp… --- ## [Is this normal for vllm serve output waiting 1 hour b´before startup](https://discuss.vllm.ai/t/is-this-normal-for-vllm-serve-output-waiting-1-hour-b-before-startup/2946)
**Author:** [@edib-isic](https://discuss.vllm.ai/u/edib-isic)\ **Replies:** 7\ **Last updated:** [September 9, 2026, 7:29am UTC](https://discuss.vllm.ai/t/is-this-normal-for-vllm-serve-output-waiting-1-hour-b-before-startup/2946 "2026-09-09T07:29:58Z")
i just started vllm serve is this the normal output of vLLM is it stuck because cannot reach it on localhost:8000 no INFO Server Started or serving at?? is this ok output?? everything seems ok $ vllm serve unsloth/… --- ## [A general auto-tuner for vLLM: Optuna, workload replay and SLO-aware optimization](https://discuss.vllm.ai/t/a-general-auto-tuner-for-vllm-optuna-workload-replay-and-slo-aware-optimization/2948)
**Author:** [@VincentG1234](https://discuss.vllm.ai/u/VincentG1234)\ **Replies:** 0\ **Last updated:** [September 8, 2026, 6:39pm UTC](https://discuss.vllm.ai/t/a-general-auto-tuner-for-vllm-optuna-workload-replay-and-slo-aware-optimization/2948 "2026-09-08T18:39:10Z")
Hi, We’ve been working on a more general auto-tuning framework for vLLM in the context of large-scale LLM inference for the French public sector: The project originally started as a fork of the OpenShift PSAP auto-tu… --- ## [vLLM TurboQuant Implementation](https://discuss.vllm.ai/t/vllm-turboquant-implementation/2944)
**Author:** [@TejasNC](https://discuss.vllm.ai/u/TejasNC)\ **Replies:** 1\ **Last updated:** [September 7, 2026, 3:36pm UTC](https://discuss.vllm.ai/t/vllm-turboquant-implementation/2944 "2026-09-07T15:36:30Z")
I came across this A First Comprehensive Study of TurboQuant: Accuracy and Performance | vLLM Blog which compares fp8 kv quantization against various TQ configs in vLLM. On looking into the code, I figure that both the … --- ## [vLLM-Ascend在银河麒麟桌面版上不开Eager就报错，开了性能暴跌，换服务器版就正常，这是系统差异导致的吗？](https://discuss.vllm.ai/t/vllm-ascend-eager/2938)
**Author:** [@oeyoews](https://discuss.vllm.ai/u/oeyoews)\ **Replies:** 1\ **Last updated:** [September 4, 2026, 6:08am UTC](https://discuss.vllm.ai/t/vllm-ascend-eager/2938 "2026-09-04T06:08:13Z")
vllm-ascend（310p) 是否支持银河麒麟桌面版操作系统， 如果不开启eager mode, 就会提示下面这个报错， 但是如果开启eager mode, 吞吐 仅仅只有 5~6 tokens/s ， 如果把 桌面版操作系统换成银河麒麟服务器版本， 不用开启eager的情况下，吞吐速度就会有成倍的提升 ant to get the accurate stacktrace, please set the environment… --- ## [请问一下最新版的vllm-ascend是否支持GLM-5.3 flash](https://discuss.vllm.ai/t/vllm-ascend-glm-5-3-flash/2931)
**Author:** [@IceWater](https://discuss.vllm.ai/u/IceWater)\ **Replies:** 1\ **Last updated:** [August 28, 2026, 8:40am UTC](https://discuss.vllm.ai/t/vllm-ascend-glm-5-3-flash/2931 "2026-08-28T08:40:12Z")
请问一下最新版的vllm-ascend是否支持GLM-5.3 flash和deepseek v4-flash 0731 --- ## [GFX1030 Support PR](https://discuss.vllm.ai/t/gfx1030-support-pr/2930)
**Author:** [@BlivionIaG](https://discuss.vllm.ai/u/BlivionIaG)\ **Replies:** 1\ **Last updated:** [August 27, 2026, 10:07pm UTC](https://discuss.vllm.ai/t/gfx1030-support-pr/2930 "2026-08-27T22:07:49Z")
Hello there, I couldn’t find a space or place to ask about where to discuss the gfx1030 pr feat(rocm): enable gfx1030 (RDNA2) as a recognized RDNA arch by BlivionIaG · Pull Request #52391 · vllm-project/vllm · GitHub R… --- ## [我用单卡跑通了deepseek-v4-flash-0731的dspark模式](https://discuss.vllm.ai/t/deepseek-v4-flash-0731-dspark/2914)
**Author:** [@ChongqingBigShark](https://discuss.vllm.ai/u/ChongqingBigShark)\ **Replies:** 1\ **Last updated:** [August 22, 2026, 11:35am UTC](https://discuss.vllm.ai/t/deepseek-v4-flash-0731-dspark/2914 "2026-08-22T11:35:14Z")
(APIServer pid=17) INFO 08-22 18:39:39 \[api\_utils.py:345\] (APIServer pid=17) INFO 08-22 18:39:39 \[api\_utils.py:345\] █ █ █▄ ▄█ (APIServer pid=17) INFO 08-22 18:39:39 \[api\_utils.py:345\] ▄▄ ▄█ █ █ … --- ## [Flash-0731 在 vLLM 0.27.1 + SM120 上 dspark 不可启动](https://discuss.vllm.ai/t/flash-0731-vllm-0-27-1-sm120-dspark/2912)
**Author:** [@ChongqingBigShark](https://discuss.vllm.ai/u/ChongqingBigShark)\ **Replies:** 1\ **Last updated:** [August 21, 2026, 2:05pm UTC](https://discuss.vllm.ai/t/flash-0731-vllm-0-27-1-sm120-dspark/2912 "2026-08-21T14:05:30Z")
vLLM 0.27.1 源码 bug——主模型和 DSpark 草稿模型共用同一个 marlin 专家类，但主模型的 named\_parameters() 里有 w13\_weight\_scale，草稿模型的却没有（参数注册不一致）。默认 DeepGEMM 下不暴露，切到 marlin 才触发。参数层面无解。——MXFP4 专家四个后端全灭： DeepGEMM：布局断言崩 Triton：源码排除 SM120 flashinfer\_cu… --- ## [Why does changing VLLM\_CACHE\_ROOT not reproduce cold-start latency?](https://discuss.vllm.ai/t/why-does-changing-vllm-cache-root-not-reproduce-cold-start-latency/2910)
**Author:** [@harry](https://discuss.vllm.ai/u/harry)\ **Replies:** 1\ **Last updated:** [August 21, 2026, 2:05pm UTC](https://discuss.vllm.ai/t/why-does-changing-vllm-cache-root-not-reproduce-cold-start-latency/2910 "2026-08-21T14:05:19Z")
I’m running a vLLM server inside a container and using VLLM\_CACHE\_ROOT to specify the cache directory. On the container’s first startup, after launching the vLLM server for the first time, the latency of the first reque… --- ## [Model path in compute\_hash() prevents torch.compile cache reuse across identical model loads](https://discuss.vllm.ai/t/model-path-in-compute-hash-prevents-torch-compile-cache-reuse-across-identical-model-loads/2908)
**Author:** [@twb1235](https://discuss.vllm.ai/u/twb1235)\ **Replies:** 1\ **Last updated:** [August 21, 2026, 2:04pm UTC](https://discuss.vllm.ai/t/model-path-in-compute-hash-prevents-torch-compile-cache-reuse-across-identical-model-loads/2908 "2026-08-21T14:04:38Z")
The model field (HuggingFace model name or local path) is currently included in ModelConfig.compute\_hash(), which means two instances serving the exact same model from different paths will compute different config\_hash v… --- ## [Mooncake prefix information in storage](https://discuss.vllm.ai/t/mooncake-prefix-information-in-storage/2904)
**Author:** [@svg](https://discuss.vllm.ai/u/svg)\ **Replies:** 3\ **Last updated:** [August 21, 2026, 5:08am UTC](https://discuss.vllm.ai/t/mooncake-prefix-information-in-storage/2904 "2026-08-21T05:08:43Z")
Is it true that In MoonCake, the prefix information disappears when it reaches the storage? Can the prefix information be preserved for use by the eviction algorithm? --- ## [Vllm是否支持加载deepseek\_ocr模型的lora适配器进行推理](https://discuss.vllm.ai/t/vllm-deepseek-ocr-lora/2161)
**Author:** [@sanjer](https://discuss.vllm.ai/u/sanjer)\ **Replies:** 3\ **Last updated:** [August 21, 2026, 12:14am UTC](https://discuss.vllm.ai/t/vllm-deepseek-ocr-lora/2161 "2026-08-21T00:14:57Z")
使用ms-swift对deepseek\_ocr模型进行lora微调得到lora适配器（正常的，lora未合并），但接下来使用vllm最新版本v0.13.0来加载这个适配器推理时报错了 self.model = self.load\_lora\_model( (EngineCore\_DP0 pid=10754) ERROR 12-28 14:53:45 \[core.py:866\] ^^^^^^^^^^^^… --- ## [Dlash2 無法使用 在 vllm 0.27.1](https://discuss.vllm.ai/t/dlash2-vllm-0-27-1/2894)
**Author:** [@kuolung1](https://discuss.vllm.ai/u/kuolung1)\ **Replies:** 1\ **Last updated:** [August 19, 2026, 8:48am UTC](https://discuss.vllm.ai/t/dlash2-vllm-0-27-1/2894 "2026-08-19T08:48:10Z")
vllm serve Qwen/Qwen3.8-27B –speculative-config ‘{ “method”: “dflash”, “model”: “incoai/Qwen3.8-27B-DFlash2”, “num\_speculative\_tokens”: 7 }’ vllm 0.27.1 (APIServer pid=1116141) File “/home/user/vLLM/env0271/lib/… --- ## [How can I choose custom backend in torch.compile](https://discuss.vllm.ai/t/how-can-i-choose-custom-backend-in-torch-compile/2889)
**Author:** [@PrimedErwin](https://discuss.vllm.ai/u/PrimedErwin)\ **Replies:** 3\ **Last updated:** [August 18, 2026, 9:04am UTC](https://discuss.vllm.ai/t/how-can-i-choose-custom-backend-in-torch-compile/2889 "2026-08-18T09:04:21Z")
in vllm/compilation/backends.py, function make\_compiler() says that we can “Using custom backend:”, but how can I use my own compiler backend? I also noticed that wrapper.py in the same dir may help. What exactly should … --- ## [Expert Parallelism All-to-All Communication without NVLink and DeepEP](https://discuss.vllm.ai/t/expert-parallelism-all-to-all-communication-without-nvlink-and-deepep/2421)
**Author:** [@Tianchen2001](https://discuss.vllm.ai/u/Tianchen2001)\ **Replies:** 4\ **Last updated:** [August 17, 2026, 9:12pm UTC](https://discuss.vllm.ai/t/expert-parallelism-all-to-all-communication-without-nvlink-and-deepep/2421 "2026-08-17T21:12:04Z")
Hi, I would like to deploy Deepseek-V2-Lite on my computer with 2 NVIDIA 3090 GPUs. There is not P2P communication channel between GPUs. By default, it seems to activate all-reduce-scatter for MoE layer. However, is ther… --- ## [FLASHINFER is FASTER then FA2 on Ampere hardware](https://discuss.vllm.ai/t/flashinfer-is-faster-then-fa2-on-ampere-hardware/2883)
**Author:** [@Frenzy\_Biscuit](https://discuss.vllm.ai/u/Frenzy_Biscuit)\ **Replies:** 5\ **Last updated:** [August 16, 2026, 7:32pm UTC](https://discuss.vllm.ai/t/flashinfer-is-faster-then-fa2-on-ampere-hardware/2883 "2026-08-16T19:32:35Z")
Hardware: 4x3090 To recap on my issues with FA2: MTP mode on either gemma 4 31B IT or Qwen 3.8 27B was slow, especially at longer contexts; Disabling MTP resolved the slowness, but was still slower then expected on ra… [Next page](https://discuss.vllm.ai/latest.md?page=1)