# \[RFC & PR Stack\] Application-Directed Prefix Checkpoints for Hybrid GDN/Mamba (Up to 7.6x Speedup on L40S) **URL:** \ **Category:** General\ **Created:** [September 9, 2026, 2:20am UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949 "2026-09-09T02:20:11Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![nicholaskh-ai](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/nicholaskh-ai/32/1718_2.png) [@nicholaskh-ai](https://discuss.vllm.ai/u/nicholaskh-ai)\ **Post date:** [September 9, 2026, 2:20am UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949/1 "2026-09-09T02:20:11Z")
Hi vLLM Community & Maintainers, We have proposed an RFC and a 3-part PR stack introducing **Application-Directed Prefix Checkpointing and Batched Grouped Prefill Execution** for hybrid GDN/Mamba architectures (such as Qwen3.5 35B-A3B) in the V1 engine. ### Links - **RFC Discussion** : [Issue #55697](https://github.com/vllm-project/vllm/issues/55697) - **Part 1 (Frontend & Input Token Parsing)**: [PR #55873](https://github.com/vllm-project/vllm/pull/55873) - **Part 2 (Scheduler Coordination & State Machine)**: PR #55875 - **Part 3 (Batched Phased GDN Kernel & Microbenchmark)**: PR #55876 * * * ### Motivation: The Boundary Dilemma in Recurrent Prefix Caching In hybrid models like Qwen3.5, recurrent states (e.g. [H, D\_k, D\_v] per layer, ~512KB per layer) have a significantly larger memory footprint than single Attention KV blocks. Because of this, recurrent architectures can realistically afford only **1 or 2 discrete state checkpoints** per sequence. From an engine-internal perspective, it is virtually impossible to heuristically predict where that single, high-value checkpoint boundary should be placed (especially with variable-length multimodal image token expansion). Conversely, from t he business application layer (e.g., 1 shared anchor item vs. N candidate queries in catalog deduplication, agent scratchpads, or multi-turn reasoning), this boundary is immediately obvious and trivial to locate. * * * ### What We Implemented 1. **Input-Side Control Marker (`<|mamba_checkpoint|>`)**: Applications declare semantic prefix boundaries in their prompt. The token is dynamically registered, parsed, and stripped by `InputProcessor` with multimodal placeholder offset calibration (zero side-effects on model generation logits). 2. **Scheduler Dependency State Machine** : - Producer prefill chunks are truncated exactly at the checkpoint position. - The checkpoint block hash is guarded with an unready lock in `BlockPool`. Subsequent consumers are deferred to `step_skipped_waiting` without blocking unrelated requests, and woken up atomically once the Producer completes. - Supports same-step Producer/Consumer prefix block inheritance when scheduled concurrently. 3. **Batched Two-Phase Grouped Prefill Execution** : Splits grouped prefill into batched Producer and Consumer phases. In Phase 1, Producers run in a single packed kernel invocation. In Phase 2, Consumers read the shared checkpoint rows via `index_select`, copy them to private destination sl ots with `index_copy_`, and compute suffixes concurrently in a second packed kernel invocation. * * * ### Benchmark Results (NVIDIA L40S, PyTorch 2.11.0+cu130) Measured on an NVIDIA L40S GPU using `benchmarks/kernels/benchmark_grouped_gdn_prefill.py`: | Workload (Producers, Consumers) | Tokens | Legacy Serial (us) | Batched Phased (us) | Speedup | P10 ~ P90 (us) | | --- | --- | --- | --- | --- | --- | | **1P + 1C** (single candidate) | 576 | 2,146.41 | **1,548.13** | **1.39x** | 1,534 ~ 1,562 | | **1P + 2C** | 640 | 2,876.00 | **1,546.33** | **1.86x** | 1,532 ~ 1,569 | | **1P + 4C** | 768 | 4,245.61 | **1,545.72** | **2.75x** | 1,532 ~ 1,565 | | **1P + 8C** (typical batch) | 1,024 | 7,055.63 | **1,582.38** | **4.46x** | 1,569 ~ 1,594 | | **1P + 16C** (wide candidate batch) | 1,536 | 12,698.95 | **1,674.81** | **7.58x** | 1,660 ~ 1,716 | - Latency is near-flat (O(N) to O(1)): as consumers scale from 1 to 16, batched latency only increases by ~8.4% (1.54ms to 1.67ms). - At 16 consumers, latency drops by **86.8%** , yielding a **7.58x speedup**. We would love to get your feedback and thoughts on this design! ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 9, 2026, 2:20am UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949/2 "2026-09-09T02:20:32Z")
The Application-Directed Prefix Checkpointing mechanism allows applications to explicitly declare semantic prefix boundaries (e.g., with a `<|mamba_checkpoint|>` token) for hybrid GDN/Mamba models in vLLM, enabling the engine to cache and reuse recurrent state at exactly the right position for high-throughput, multi-request workloads. This approach solves the “boundary dilemma” where the engine cannot heuristically determine the optimal checkpoint location, especially with variable-length multimodal prompts, but the application knows it precisely. The scheduler coordinates Producer-Consumer prefill, eliminating redundant computation and enabling up to 7.6x speedup in batched scenarios, with near-flat latency as batch size increases, and zero accuracy loss ([see RFC #55697](https://github.com/vllm-project/vllm/issues/55697)). This design is implemented in a 3-part PR stack: frontend token parsing, scheduler state machine, and a batched grouped GDN kernel. The mechanism is multimodal-aware, ensures memory efficiency (only one physical checkpoint block per sequence), and supports atomic wakeup of consumers after the producer checkpoint is ready. Benchmarks on NVIDIA L40S show dramatic throughput and latency improvements for 1-to-N candidate matching workloads ([see RFC #55697](https://github.com/vllm-project/vllm/issues/55697)). Would you like a step-by-step breakdown of the scheduler and kernel changes? Sources: - [RFC: Application-Directed Prefix Checkpoints for Mamba / Hybrid Prefix Caching](https://github.com/vllm-project/vllm/issues/55697) Hint: Mention @Herald in the post for followups. ---
**Author:** ![nicholaskh-ai](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/nicholaskh-ai/32/1718_2.png) [@nicholaskh-ai](https://discuss.vllm.ai/u/nicholaskh-ai)\ **Post date:** [September 9, 2026, 4:32pm UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949/3 "2026-09-09T16:32:32Z")
@Herald Please help index this RFC and PR stack (#55873, #55875, #55876) for maintainer review. Thanks! ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 9, 2026, 4:32pm UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949/4 "2026-09-09T16:32:45Z")
The RFC and PR stack for “Application-Directed Prefix Checkpoints for Mamba / Hybrid Prefix Caching” consists of: - RFC: [#55697](https://github.com/vllm-project/vllm/issues/55697) — describes the motivation, design, and production results for application-directed prefix checkpointing in hybrid SSM/Attention models. - PR #55873: [Frontend & Input Processing](https://github.com/vllm-project/vllm/pull/55873) — adds CLI/engine options, checkpoint token registration, and input processing. - PR #55875: [Scheduler Coordination & State Machine](https://github.com/vllm-project/vllm/pull/55875) — implements scheduler checkpoint coordination, chunk splitting, and the unready/ready state machine. - PR #55876: [Worker & Kernel Grouped Prefill Execution](https://github.com/vllm-project/vllm/pull/55876) — introduces batched two-phase grouped prefill execution, delivering up to 7.58x speedup. All three PRs are open and depend on each other, implementing the RFC in progressive, test-covered layers. Would you like a step-by-step summary of the technical design or review priorities? Sources: - [#55697 RFC: Application-Directed Prefix Checkpoints for Mamba / Hybrid Prefix Caching](https://github.com/vllm-project/vllm/issues/55697) - [PR #55873: Frontend & Input Processing](https://github.com/vllm-project/vllm/pull/55873) - [PR #55875: Scheduler Coordination & State Machine](https://github.com/vllm-project/vllm/pull/55875) - [PR #55876: Worker & Kernel Grouped Prefill Execution](https://github.com/vllm-project/vllm/pull/55876) Hint: Mention @Herald in the post for followups. ---
**Author:** ![nicholaskh-ai](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/nicholaskh-ai/32/1718_2.png) [@nicholaskh-ai](https://discuss.vllm.ai/u/nicholaskh-ai)\ **Post date:** [September 9, 2026, 4:35pm UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949/5 "2026-09-09T16:35:25Z")
@Herald Yes, please highlight the review priorities for the maintainers. Our recommended review order: 1. **PR #55873 (Part 1 - Frontend & InputProcessor)**: Self-contained configuration, CLI options, and token parsing/offset shifting. Lowest review risk, ready for initial merge. 2. **PR #55875 (Part 2 - Scheduler Coordination)**: V1 scheduler chunk truncation at checkpoint boundaries and unready/ready state machine. 3. **PR #55876 (Part 3 - Batched Phased GDN Kernel)**: Two-phase prefill execution delivering up to **7.58x speedup** on NVIDIA L40S. Could a maintainer please help add the `ready` label to PR #55873 to kick off CI? Thanks! ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 9, 2026, 4:35pm UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949/6 "2026-09-09T16:35:43Z")
Recommended review priorities: 1. **PR #55873 (Part 1 - Frontend & InputProcessor):** Self-contained config, CLI options, and token parsing; lowest risk, ready for initial merge and CI. 2. **PR #55875 (Part 2 - Scheduler Coordination):** Scheduler chunk truncation at checkpoint boundaries and unready/ready state machine. 3. **PR #55876 (Part 3 - Batched Phased GDN Kernel):** Two-phase prefill execution, up to 7.58x speedup on NVIDIA L40S. Request: Please add the `ready` label to [PR #55873](https://github.com/vllm-project/vllm/pull/55873) to start CI, as it is the lowest risk and unblocks the stack. Would you like a step-by-step rationale for this order? Sources: - [PR #55873](https://github.com/vllm-project/vllm/pull/55873) - [RFC Issue #55697](https://github.com/vllm-project/vllm/issues/55697) Hint: Mention @Herald in the post for followups. ---
**Author:** ![nicholaskh-ai](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/nicholaskh-ai/32/1718_2.png) [@nicholaskh-ai](https://discuss.vllm.ai/u/nicholaskh-ai)\ **Post date:** [September 15, 2026, 12:59pm UTC](https://discuss.vllm.ai/t/rfc-pr-stack-application-directed-prefix-checkpoints-for-hybrid-gdn-mamba-up-to-7-6x-speedup-on-l40s/2949/7 "2026-09-15T12:59:02Z")
> [@RunLLM](#): > > add the `ready` label to [PR #55873](https://github.com/vllm-project/vllm/pull/55873) to start CI @Herald How can I add the `ready` label to [PR #55873](https://github.com/vllm-project/vllm/pull/55873) to start CI?