# Features **URL:** https://discuss.vllm.ai/c/features/13.md [Latest](https://discuss.vllm.ai/latest.md) · [Categories](https://discuss.vllm.ai/categories.md) --- ## [About the Features category](https://discuss.vllm.ai/t/about-the-features-category/80)
**Author:** [@youkaichao](https://discuss.vllm.ai/u/youkaichao)\ **Replies:** 0
--- ## [Vllm\_gguf\_plugin: iq and k quant kernel improvements WIP](https://discuss.vllm.ai/t/vllm-gguf-plugin-iq-and-k-quant-kernel-improvements-wip/3056)
**Author:** [@marked23](https://discuss.vllm.ai/u/marked23)\ **Replies:** 1\ **Last updated:** [October 14, 2026, 8:05pm UTC](https://discuss.vllm.ai/t/vllm-gguf-plugin-iq-and-k-quant-kernel-improvements-wip/3056 "2026-10-14T20:05:08Z")
The robot and I are working through the tests for vllm\_gguf\_plugin. Several of the IQ and K quant kernels failed tests on my gfx1100 GPUs. Here’s an example of what the robot did to iq1\_s.py. The current code nests t… --- ## [Allow agent to change speculative\_tokens in each request](https://discuss.vllm.ai/t/allow-agent-to-change-speculative-tokens-in-each-request/2997)
**Author:** [@alindzon](https://discuss.vllm.ai/u/alindzon)\ **Replies:** 1\ **Last updated:** [September 24, 2026, 3:04am UTC](https://discuss.vllm.ai/t/allow-agent-to-change-speculative-tokens-in-each-request/2997 "2026-09-24T03:04:40Z")
It would be great if a request could fine tune num speculative\_tokens to be fewer for thinking work and higher for coding work. On engine init specifie the max, and on requests allow a lower value such as { “extra\_body… --- ## [Decode KV writeback: can multi-turn chat preserve cache reuse with default templates?](https://discuss.vllm.ai/t/decode-kv-writeback-can-multi-turn-chat-preserve-cache-reuse-with-default-templates/2955)
**Author:** [@lhecust](https://discuss.vllm.ai/u/lhecust)\ **Replies:** 1\ **Last updated:** [September 10, 2026, 1:50am UTC](https://discuss.vllm.ai/t/decode-kv-writeback-can-multi-turn-chat-preserve-cache-reuse-with-default-templates/2955 "2026-09-10T01:50:44Z")
Hi vLLM community, I would like to understand the practical benefits of Decode-phase KV writeback when server-side chat templates change the token sequence between conversation turns. This relates to the Decode-Phase K… --- ## [vLLM TurboQuant Implementation](https://discuss.vllm.ai/t/vllm-turboquant-implementation/2944)
**Author:** [@TejasNC](https://discuss.vllm.ai/u/TejasNC)\ **Replies:** 1\ **Last updated:** [September 7, 2026, 3:36pm UTC](https://discuss.vllm.ai/t/vllm-turboquant-implementation/2944 "2026-09-07T15:36:30Z")
I came across this A First Comprehensive Study of TurboQuant: Accuracy and Performance | vLLM Blog which compares fp8 kv quantization against various TQ configs in vLLM. On looking into the code, I figure that both the … --- ## [Customize prefix caching](https://discuss.vllm.ai/t/customize-prefix-caching/2864)
**Author:** [@thiomajid](https://discuss.vllm.ai/u/thiomajid)\ **Replies:** 1\ **Last updated:** [August 9, 2026, 9:59pm UTC](https://discuss.vllm.ai/t/customize-prefix-caching/2864 "2026-08-09T21:59:59Z")
Hi everyone, I have been tinkering with LLM serving lately and I was wondering if it was possible to provide a custom implementation for prefix caching when serving a model in the mode that accepts input embeddings rathe… --- ## [Sparse attention (e.g., H2O)](https://discuss.vllm.ai/t/sparse-attention-e-g-h2o/2745)
**Author:** [@mseeger](https://discuss.vllm.ai/u/mseeger)\ **Replies:** 2\ **Last updated:** [July 1, 2026, 4:28pm UTC](https://discuss.vllm.ai/t/sparse-attention-e-g-h2o/2745 "2026-07-01T16:28:39Z")
I’d like to implement sparse attention KV cache policies (such as H2O) in vLLM. One issue I am running into is that a page (or block) needs to extend over all heads of a model. In H2O or other sparse attention methods, … --- ## [Is there a hook/flag to capture activation statistics during inference for use with llm-compressor AWQ?](https://discuss.vllm.ai/t/is-there-a-hook-flag-to-capture-activation-statistics-during-inference-for-use-with-llm-compressor-awq/2696)
**Author:** [@ManikandanB](https://discuss.vllm.ai/u/ManikandanB)\ **Replies:** 3\ **Last updated:** [June 4, 2026, 5:43pm UTC](https://discuss.vllm.ai/t/is-there-a-hook-flag-to-capture-activation-statistics-during-inference-for-use-with-llm-compressor-awq/2696 "2026-06-04T17:43:39Z")
Context AWQ quantization via llm-compressor requires activation statistics (per-channel magnitudes) collected during calibration. Typically this uses a small proxy dataset, which may not represent real production traffic… --- ## [Dose vllm support Qwen3.5 pd disaggregation with Mooncake?](https://discuss.vllm.ai/t/dose-vllm-support-qwen3-5-pd-disaggregation-with-mooncake/2668)
**Author:** [@dingfen](https://discuss.vllm.ai/u/dingfen)\ **Replies:** 1\ **Last updated:** [May 28, 2026, 10:13am UTC](https://discuss.vllm.ai/t/dose-vllm-support-qwen3-5-pd-disaggregation-with-mooncake/2668 "2026-05-28T10:13:04Z")
I ran a vllm server with model=Qwen3.5-397B on 0.20.1 vllm, and this is my server command: vllm serve $model --trust-remote-code -tp 8 --port 8020 --kv-transfer-config ‘{“kv\_connector”:“MooncakeConnector”,“kv\_role”:“kv\_… --- ## [Why Does Decode Forward on PP Stage 0 Appear to Precede Prefill Forward on PP Stage 1 for the Same Request?](https://discuss.vllm.ai/t/why-does-decode-forward-on-pp-stage-0-appear-to-precede-prefill-forward-on-pp-stage-1-for-the-same-request/2661)
**Author:** [@joeyxzy](https://discuss.vllm.ai/u/joeyxzy)\ **Replies:** 1\ **Last updated:** [May 26, 2026, 1:18am UTC](https://discuss.vllm.ai/t/why-does-decode-forward-on-pp-stage-0-appear-to-precede-prefill-forward-on-pp-stage-1-for-the-same-request/2661 "2026-05-26T01:18:53Z")
Summary We instrument vLLM 0.20’s GPU execution phases using time.time\_ns() (CPU wall clock) across all PP worker processes. For a single autoregressive request running with tp=2, pp=2, we observe the following on th… --- ## [GPTQModel 能量化 GLM-5 FP16 到 INT8 吗](https://discuss.vllm.ai/t/gptqmodel-glm-5-fp16-int8/2563)
**Author:** [@XiaoDouGeGe](https://discuss.vllm.ai/u/XiaoDouGeGe)\ **Replies:** 9\ **Last updated:** [April 24, 2026, 10:02am UTC](https://discuss.vllm.ai/t/gptqmodel-glm-5-fp16-int8/2563 "2026-04-24T10:02:13Z")
1、GPTQModel 能量化 GLM-5 FP16 到 INT8 吗？ 2、GPTQModel gptqmodel-6.0.3.tar.gz 如何 offload disk，麻烦提供配置示例？ --- ## [DeepSeek MTP full cuda graph support?](https://discuss.vllm.ai/t/deepseek-mtp-full-cuda-graph-support/2548)
**Author:** [@dingfen](https://discuss.vllm.ai/u/dingfen)\ **Replies:** 3\ **Last updated:** [April 13, 2026, 8:22am UTC](https://discuss.vllm.ai/t/deepseek-mtp-full-cuda-graph-support/2548 "2026-04-13T08:22:37Z")
Hey guys! I am working on vLLM 0.17.1, and I am running DeepSeek R1 with Spec Config like that: { "method": "deepseek\_mtp", "num\_speculative\_tokens": 3, "draft\_tensor\_parallel\_size": 16 } And of cource for best … --- ## [Qwen3.5-27B-FP8 Speculative Decoding](https://discuss.vllm.ai/t/qwen3-5-27b-fp8-speculative-decoding/2447)
**Author:** [@WittyExplorer](https://discuss.vllm.ai/u/WittyExplorer)\ **Replies:** 2\ **Last updated:** [April 11, 2026, 2:02am UTC](https://discuss.vllm.ai/t/qwen3-5-27b-fp8-speculative-decoding/2447 "2026-04-11T02:02:01Z")
I want to test speculative decoding with Qwen3.5-27B-FB8. The only way that works for me is {“method”: “mtp”, “num\_speculative\_tokens”:1} . Increasing num\_speculative\_tokens to 2 results in error. Is that expected? Look… --- ## [thinking\_token\_budget silently ignored when passed via extra\_args in vLLM 0.18.0](https://discuss.vllm.ai/t/thinking-token-budget-silently-ignored-when-passed-via-extra-args-in-vllm-0-18-0/2533)
**Author:** [@corvo1837](https://discuss.vllm.ai/u/corvo1837)\ **Replies:** 1\ **Last updated:** [April 11, 2026, 1:56am UTC](https://discuss.vllm.ai/t/thinking-token-budget-silently-ignored-when-passed-via-extra-args-in-vllm-0-18-0/2533 "2026-04-11T01:56:48Z")
Hi all, We’re running Qwen/Qwen3.5-9B with thinking mode enabled on vLLM v0.18.0, using the Python API directly (not the OpenAI-compatible server). Our use case is structured JSON extraction from document images — some … --- ## [GLM 5 / Kimi k2.5 on 4 x RTX 6000 Pro](https://discuss.vllm.ai/t/glm-5-kimi-k2-5-on-4-x-rtx-6000-pro/2480)
**Author:** [@valn](https://discuss.vllm.ai/u/valn)\ **Replies:** 1\ **Last updated:** [March 22, 2026, 3:09pm UTC](https://discuss.vllm.ai/t/glm-5-kimi-k2-5-on-4-x-rtx-6000-pro/2480 "2026-03-22T15:09:41Z")
hi. Is there any way to get those models running via VLLM on such hardware? They “almost” have enough ram for AWQ or similar quants but not quite. It would be a killer feature to be able to run them. One way would be su… --- ## [Compressed Multimodal embeddings inputs](https://discuss.vllm.ai/t/compressed-multimodal-embeddings-inputs/2469)
**Author:** [@guillaume.guy](https://discuss.vllm.ai/u/guillaume.guy)\ **Replies:** 1\ **Last updated:** [March 18, 2026, 5:49pm UTC](https://discuss.vllm.ai/t/compressed-multimodal-embeddings-inputs/2469 "2026-03-18T17:49:24Z")
vLLM supports multimodal inputs through the multi\_modal\_data argument. This works well but the argument tends to be quite large. For QWEN3VL-4B with a photo of 384x384, we are talking about 2MB payload (500k floats). On… --- ## [NVFP4 Support In Attention](https://discuss.vllm.ai/t/nvfp4-support-in-attention/2457)
**Author:** [@lightzhan](https://discuss.vllm.ai/u/lightzhan)\ **Replies:** 1\ **Last updated:** [March 16, 2026, 3:11am UTC](https://discuss.vllm.ai/t/nvfp4-support-in-attention/2457 "2026-03-16T03:11:55Z")
Hi, guys I am looking through some vllm blog posts and confused on kv cache quantization. Does vllm supprot nvfp4 calculation for QK and PV calculation? Does vllm support nvfp4 kv cache storage?(in this case, kv cache… --- ## [Distributed Speculative Decoding using Ray](https://discuss.vllm.ai/t/distributed-speculative-decoding-using-ray/2368)
**Author:** [@Loris-EPFL](https://discuss.vllm.ai/u/Loris-EPFL)\ **Replies:** 3\ **Last updated:** [February 11, 2026, 7:54pm UTC](https://discuss.vllm.ai/t/distributed-speculative-decoding-using-ray/2368 "2026-02-11T19:54:20Z")
Good morning, I have a dgx spark and a rtx 5090 node, which I use in a ray cluster using pipeline parrallelism and asymetric layer assignment. I looked vLLM Speculative Decoding docs, and would like to run the speculati… --- ## [Deployment example for a qwen3 model with hybrid thinking](https://discuss.vllm.ai/t/deployment-example-for-a-qwen3-model-with-hybrid-thinking/1462)
**Author:** [@hustxiayang](https://discuss.vllm.ai/u/hustxiayang)\ **Replies:** 10\ **Last updated:** [February 4, 2026, 5:25pm UTC](https://discuss.vllm.ai/t/deployment-example-for-a-qwen3-model-with-hybrid-thinking/1462 "2026-02-04T17:25:25Z")
Could you please show me a deployment example for a qwen3 model with hybrid thinking, for example, Qwen3-30B-A3B, and example of disabling thinking --- ## [How do I precompute multimodal embeddings?](https://discuss.vllm.ai/t/how-do-i-precompute-multimodal-embeddings/2294)
**Author:** [@kevinpw](https://discuss.vllm.ai/u/kevinpw)\ **Replies:** 5\ **Last updated:** [February 2, 2026, 12:11pm UTC](https://discuss.vllm.ai/t/how-do-i-precompute-multimodal-embeddings/2294 "2026-02-02T12:11:18Z")
I want to efficiently generate embeddings with Qwen3-VL-Embedding-8B for a single image using multiple different text instructions. Embedding generation is relatively slow because the model expects instructions in the sy… --- ## [Implementing hidden state probes](https://discuss.vllm.ai/t/implementing-hidden-state-probes/2291)
**Author:** [@LiquidGunay](https://discuss.vllm.ai/u/LiquidGunay)\ **Replies:** 1\ **Last updated:** [January 30, 2026, 4:19pm UTC](https://discuss.vllm.ai/t/implementing-hidden-state-probes/2291 "2026-01-30T16:19:12Z")
There has been a lot of recent work on trying to use the hidden state of an LLM as an input to classifiers for identifying things like harmful content and jailbreaks (like Anthropic’s two-stage classifier approach). This… --- ## [Standalone draft model spec decode support in v0.x and v1](https://discuss.vllm.ai/t/standalone-draft-model-spec-decode-support-in-v0-x-and-v1/2241)
**Author:** [@luxisme](https://discuss.vllm.ai/u/luxisme)\ **Replies:** 3\ **Last updated:** [January 20, 2026, 8:31am UTC](https://discuss.vllm.ai/t/standalone-draft-model-spec-decode-support-in-v0-x-and-v1/2241 "2026-01-20T08:31:22Z")
Standalone draft model is removed in v0.x releases (x\>10): Speculative Decoding - vLLM And v1 does not support this either. Can I know why this is not supported anymore? --- ## [How to get kv cache value from vllm](https://discuss.vllm.ai/t/how-to-get-kv-cache-value-from-vllm/2100)
**Author:** [@chx725](https://discuss.vllm.ai/u/chx725)\ **Replies:** 5\ **Last updated:** [January 19, 2026, 3:48am UTC](https://discuss.vllm.ai/t/how-to-get-kv-cache-value-from-vllm/2100 "2026-01-19T03:48:13Z")
I want to catch the kv cache from LLM model to do some other analysis, but I don’t know how to do it. --- ## [Is there a plan for EVS to support Qwen3VL in response to the issue of sparse video tokens?](https://discuss.vllm.ai/t/is-there-a-plan-for-evs-to-support-qwen3vl-in-response-to-the-issue-of-sparse-video-tokens/2215)
**Author:** [@XSH](https://discuss.vllm.ai/u/XSH)\ **Replies:** 1\ **Last updated:** [January 13, 2026, 9:31pm UTC](https://discuss.vllm.ai/t/is-there-a-plan-for-evs-to-support-qwen3vl-in-response-to-the-issue-of-sparse-video-tokens/2215 "2026-01-13T21:31:35Z")
Currently, EVS supports QWen2.5vl. Are there plans to support QWen3vl? When trying to upgrade EVS to QWen3VL, the get\_video\_replacement\_qwen3vl method in the qwen3\_vl.py file first calculates the number of video\_token\_ids… --- ## [Exposing KV cache for recomposition / reuse beyond prefix caching?](https://discuss.vllm.ai/t/exposing-kv-cache-for-recomposition-reuse-beyond-prefix-caching/2208)
**Author:** [@CharlessJYH](https://discuss.vllm.ai/u/CharlessJYH)\ **Replies:** 1\ **Last updated:** [January 13, 2026, 9:30pm UTC](https://discuss.vllm.ai/t/exposing-kv-cache-for-recomposition-reuse-beyond-prefix-caching/2208 "2026-01-13T21:30:29Z")
I have a question about KV cache reuse. Currently, I understand that enable\_prefix\_caching only supports reuse when the prompt prefix is exactly the same. My use case is slightly different: I want to reuse KV cache fr… --- ## [Why I feel cuda-kernel marlin run not fast?](https://discuss.vllm.ai/t/why-i-feel-cuda-kernel-marlin-run-not-fast/2197)
**Author:** [@cyk2018](https://discuss.vllm.ai/u/cyk2018)\ **Replies:** 5\ **Last updated:** [January 9, 2026, 5:31pm UTC](https://discuss.vllm.ai/t/why-i-feel-cuda-kernel-marlin-run-not-fast/2197 "2026-01-09T17:31:22Z")
Hello, Every one. When I run the Qwen3-32B-awq model with vllm, I found the profile result show that the marlin kernel is slow, I think this operator is memory bound and I calculate the memory unit rate find almost slow … --- ## [Can reasoning\_effort parameter not ne used in vllm implementation via python?](https://discuss.vllm.ai/t/can-reasoning-effort-parameter-not-ne-used-in-vllm-implementation-via-python/2176)
**Author:** [@JananiShruthi](https://discuss.vllm.ai/u/JananiShruthi)\ **Replies:** 1\ **Last updated:** [January 2, 2026, 10:20am UTC](https://discuss.vllm.ai/t/can-reasoning-effort-parameter-not-ne-used-in-vllm-implementation-via-python/2176 "2026-01-02T10:20:39Z")
I am currently using vllm to infer from the model gpt-oss-20b. I want to keep the model’s reasoning ability to low. I read that vllm support a parameter called reasoning\_effort. Can I use that parameter if I am using vll… --- ## [Understanding vllm kv cache](https://discuss.vllm.ai/t/understanding-vllm-kv-cache/2061)
**Author:** [@rahulraman1604](https://discuss.vllm.ai/u/rahulraman1604)\ **Replies:** 5\ **Last updated:** [December 1, 2025, 9:50pm UTC](https://discuss.vllm.ai/t/understanding-vllm-kv-cache/2061 "2025-12-01T21:50:56Z")
Hoping to get a high level overview based on the vllm docs on 0.11.2 and the github code, how is the KV cache being managed. Explain from physical GPU blocks to virtual blocks how is it being managed, what are the layers… --- ## [Has anyone successfully run DBO in a single node multi card environment？](https://discuss.vllm.ai/t/has-anyone-successfully-run-dbo-in-a-single-node-multi-card-environment/2054)
**Author:** [@bleedingfight](https://discuss.vllm.ai/u/bleedingfight)\ **Replies:** 1\ **Last updated:** [December 1, 2025, 7:35am UTC](https://discuss.vllm.ai/t/has-anyone-successfully-run-dbo-in-a-single-node-multi-card-environment/2054 "2025-12-01T07:35:14Z")
I want to try DBO to check the running effect of MOE model, and then I will try to run the case given in the demo first. So I installed DeepEP(python setup.py install not error)。I have tested: \[nopass\] python tests/tes… --- ## [EPLB behavior in elastic scaling](https://discuss.vllm.ai/t/eplb-behavior-in-elastic-scaling/2047)
**Author:** [@xeonliu](https://discuss.vllm.ai/u/xeonliu)\ **Replies:** 21\ **Last updated:** [November 28, 2025, 10:11am UTC](https://discuss.vllm.ai/t/eplb-behavior-in-elastic-scaling/2047 "2025-11-28T10:11:43Z")
class DPLBAsyncMPClient(DPAsyncMPClient): async def \_scale\_up\_elastic\_ep(self, cur\_data\_parallel\_size: int, new\_data\_parallel\_size: int) -\> None: """Scale up the data parall… [Next page](https://discuss.vllm.ai/c/features/13.md?page=1)