We have proposed an RFC and a 3-part PR stack introducing Application-Directed Prefix Checkpointing and Batched Grouped Prefill Execution for hybrid GDN/Mamba architectures (such as Qwen3.5 35B-A3B) in the V1 engine.
Part 1 (Frontend & Input Token Parsing): PR #55873
Part 2 (Scheduler Coordination & State Machine): PR #55875
Part 3 (Batched Phased GDN Kernel & Microbenchmark): PR #55876
Motivation: The Boundary Dilemma in Recurrent Prefix Caching
In hybrid models like Qwen3.5, recurrent states (e.g. [H, D_k, D_v] per layer, ~512KB per layer) have a significantly larger memory footprint than single Attention KV blocks. Because of this, recurrent architectures can realistically afford
only 1 or 2 discrete state checkpoints per sequence.
From an engine-internal perspective, it is virtually impossible to heuristically predict where that single, high-value checkpoint boundary should be placed (especially with variable-length multimodal image token expansion). Conversely, from t
he business application layer (e.g., 1 shared anchor item vs. N candidate queries in catalog deduplication, agent scratchpads, or multi-turn reasoning), this boundary is immediately obvious and trivial to locate.
What We Implemented
Input-Side Control Marker (<|mamba_checkpoint|>):
Applications declare semantic prefix boundaries in their prompt. The token is dynamically registered, parsed, and stripped by InputProcessor with multimodal placeholder offset calibration (zero side-effects on model generation logits).
Scheduler Dependency State Machine:
Producer prefill chunks are truncated exactly at the checkpoint position.
The checkpoint block hash is guarded with an unready lock in BlockPool. Subsequent consumers are deferred to step_skipped_waiting without blocking unrelated requests, and woken up atomically once the Producer completes.
Supports same-step Producer/Consumer prefix block inheritance when scheduled concurrently.
Batched Two-Phase Grouped Prefill Execution:
Splits grouped prefill into batched Producer and Consumer phases. In Phase 1, Producers run in a single packed kernel invocation. In Phase 2, Consumers read the shared checkpoint rows via index_select, copy them to private destination sl
ots with index_copy_, and compute suffixes concurrently in a second packed kernel invocation.
The Application-Directed Prefix Checkpointing mechanism allows applications to explicitly declare semantic prefix boundaries (e.g., with a <|mamba_checkpoint|> token) for hybrid GDN/Mamba models in vLLM, enabling the engine to cache and reuse recurrent state at exactly the right position for high-throughput, multi-request workloads. This approach solves the “boundary dilemma” where the engine cannot heuristically determine the optimal checkpoint location, especially with variable-length multimodal prompts, but the application knows it precisely. The scheduler coordinates Producer-Consumer prefill, eliminating redundant computation and enabling up to 7.6x speedup in batched scenarios, with near-flat latency as batch size increases, and zero accuracy loss (see RFC #55697).
This design is implemented in a 3-part PR stack: frontend token parsing, scheduler state machine, and a batched grouped GDN kernel. The mechanism is multimodal-aware, ensures memory efficiency (only one physical checkpoint block per sequence), and supports atomic wakeup of consumers after the producer checkpoint is ready. Benchmarks on NVIDIA L40S show dramatic throughput and latency improvements for 1-to-N candidate matching workloads (see RFC #55697). Would you like a step-by-step breakdown of the scheduler and kernel changes?
All three PRs are open and depend on each other, implementing the RFC in progressive, test-covered layers. Would you like a step-by-step summary of the technical design or review priorities?
PR #55873 (Part 1 - Frontend & InputProcessor): Self-contained config, CLI options, and token parsing; lowest risk, ready for initial merge and CI.
PR #55875 (Part 2 - Scheduler Coordination): Scheduler chunk truncation at checkpoint boundaries and unready/ready state machine.
PR #55876 (Part 3 - Batched Phased GDN Kernel): Two-phase prefill execution, up to 7.58x speedup on NVIDIA L40S.
Request: Please add the ready label to PR #55873 to start CI, as it is the lowest risk and unblocks the stack.
Would you like a step-by-step rationale for this order?