# Who ensures inputs for middle piecewise graph are static buffers?

**URL:** <https://discuss.vllm.ai/t/who-ensures-inputs-for-middle-piecewise-graph-are-static-buffers/1548>\
**Category:** torch.compile integration\
**Created:** [September 4, 2025, 1:33am UTC](https://discuss.vllm.ai/t/who-ensures-inputs-for-middle-piecewise-graph-are-static-buffers/1548 "2025-09-04T01:33:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![gx16377](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/gx16377/32/686_2.png) [@gx16377](https://discuss.vllm.ai/u/gx16377)\
**Post date:** [September 4, 2025, 1:33am UTC](https://discuss.vllm.ai/t/who-ensures-inputs-for-middle-piecewise-graph-are-static-buffers/1548/1 "2025-09-04T01:33:42Z")

</div>

Hi, vLLM Team, I’m curious about piecewise graph mechanism.

I just run into a question, CUDAGraphWrapper(or previous CudaPiecewiseBackend) needs caller ensure input buffers are same address during capture and replay, but I failed to figure out who did this.

copy\_and\_call in VllmBackend seems related but it only works for inputs of whole module(which are also inputs of 1st graph). What about other graphs in the middle of whole module(such as 2nd,3rd… pieces), especially for successors of splitting\_ops?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [September 4, 2025, 1:34am UTC](https://discuss.vllm.ai/t/who-ensures-inputs-for-middle-piecewise-graph-are-static-buffers/1548/2 "2025-09-04T01:34:10Z")

</div>

In vLLM’s piecewise CUDA graph mechanism, the input buffer address consistency for each subgraph (piece) is managed by the compiler backend. Specifically, for each piece after splitting (e.g., after attention ops), vLLM pre-allocates persistent input buffers for all symbolic-shaped inputs (like input\_ids, positions, etc.) and ensures that, during both capture and replay, the same memory addresses are used. The copy-and-call logic in the backend copies runtime inputs into these pre-allocated buffers before invoking the subgraph, so all pieces—including those after splitting\_ops—receive inputs at the same addresses for every run. This mechanism is handled internally by the backend and is not limited to just the first piece; it applies to all submodules created by the graph split, ensuring address consistency for every piecewise cudagraph capture and replay according to [vllm/compilation/backends.py](https://docs.vllm.ai/en/latest/api/vllm/compilation/backends.html#__code_57), [CUDAPiecewiseBackend docs](https://docs.vllm.ai/en/latest/api/vllm/compilation/cuda_piecewise_backend.html#vllm.compilation.cuda_piecewise_backend.CUDAPiecewiseBackend.__init__), and [design/torch\_compile](https://docs.vllm.ai/en/latest/design/torch_compile.html#__span-1-19).

Would you like a step-by-step breakdown of how the persistent buffer and copy mechanism works for all subgraphs?

Sources:

- [vllm/compilation/backends.py](https://docs.vllm.ai/en/latest/api/vllm/compilation/backends.html#__code_57)
- [CUDAPiecewiseBackend docs](https://docs.vllm.ai/en/latest/api/vllm/compilation/cuda_piecewise_backend.html#vllm.compilation.cuda_piecewise_backend.CUDAPiecewiseBackend. __init__ )
- [design/torch\_compile](https://docs.vllm.ai/en/latest/design/torch_compile.html#__span-1-19)