# Disagg Prefill timeout

**URL:** <https://discuss.vllm.ai/t/disagg-prefill-timeout/1037>\
**Category:** Disaggregated Prefilling\
**Created:** [July 7, 2025, 9:29pm UTC](https://discuss.vllm.ai/t/disagg-prefill-timeout/1037 "2025-07-07T21:29:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![PratishthaGaur](https://avatars.discourse-cdn.com/v4/letter/p/9e8a1a/32.png) [@PratishthaGaur](https://discuss.vllm.ai/u/PratishthaGaur)\
**Post date:** [July 7, 2025, 9:29pm UTC](https://discuss.vllm.ai/t/disagg-prefill-timeout/1037/1 "2025-07-07T21:29:36Z")

</div>

I am trying to run the benchmarks/disagg\_benchmark but it seems like the requests are sent but not received. I get this error once the buffer is full.

**Error:**  
Traceback (most recent call last):  
File “/opt/conda/envs/py\_3.12/lib/python3.12/threading.py”, line 1075, in \_bootstrap\_inner  
self.run()  
File “/opt/conda/envs/py\_3.12/lib/python3.12/threading.py”, line 1012, in run  
self.\_target(\*self.\_args, \*\*self.\_kwargs)  
File “/var/lib/jenkins/vllm/vllm/distributed/kv\_transfer/kv\_lookup\_buffer/simple\_buffer.py”, line 181, in drop\_select\_handler  
raise e  
File “/var/lib/jenkins/vllm/vllm/distributed/kv\_transfer/kv\_lookup\_buffer/simple\_buffer.py”, line 140, in drop\_select\_handler  
signal = self.signal\_pipe.recv\_tensor()  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/var/lib/jenkins/vllm/vllm/distributed/kv\_transfer/kv\_pipe/pynccl\_pipe.py”, line 269, in recv\_tensor  
raise e  
File “/var/lib/jenkins/vllm/vllm/distributed/kv\_transfer/kv\_pipe/pynccl\_pipe.py”, line 262, in recv\_tensor  
tensor = future.result()  
^^^^^^^^^^^^^^^  
File “/opt/conda/envs/py\_3.12/lib/python3.12/concurrent/futures/\_base.py”, line 456, in result  
return self.\_\_get\_result()  
^^^^^^^^^^^^^^^^^^^  
File “/opt/conda/envs/py\_3.12/lib/python3.12/concurrent/futures/\_base.py”, line 401, in \_\_get\_result  
raise self.\_exception  
File “/opt/conda/envs/py\_3.12/lib/python3.12/concurrent/futures/thread.py”, line 59, in run  
result = self.fn(\*self.args, \*\*self.kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/var/lib/jenkins/vllm/vllm/distributed/kv\_transfer/kv\_pipe/pynccl\_pipe.py”, line 192, in \_recv\_impl  
metadata = self.\_recv\_metadata()  
^^^^^^^^^^^^^^^^^^^^^  
File “/var/lib/jenkins/vllm/vllm/distributed/kv\_transfer/kv\_pipe/pynccl\_pipe.py”, line 167, in \_recv\_metadata  
return self.group.recv\_obj(self.target\_rank\_for\_recv)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File “/var/lib/jenkins/vllm/vllm/distributed/utils.py”, line 169, in recv\_obj  
self.store.get(  
torch.distributed.DistStoreError: wait timeout after 300000ms, keys: /send\_to/0/0

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [July 7, 2025, 9:30pm UTC](https://discuss.vllm.ai/t/disagg-prefill-timeout/1037/2 "2025-07-07T21:30:30Z")

</div>

This error indicates a timeout in the KV receiving thread, typically caused by the buffer being full and the decode instance not consuming requests fast enough. This is a known issue in vLLM’s disaggregated prefill mode, especially under high QPS or when the `kv_buffer_size` is small. The prefill instance may batch too many requests, causing some to fail and block the rest, leading to a deadlock where the buffer fills up and no further requests are processed. Setting `--max-num-seqs` in the prefill arguments can help control batch size and mitigate this issue, as confirmed by user reports and maintainers in the vLLM repository. There is no official fix yet, but this workaround is recommended for now. For more details, see the discussion in [vllm/issues/11247](https://github.com/vllm-project/vllm/issues/11247) and [vllm/issues/14193](https://github.com/vllm-project/vllm/issues/14193).

Would you like a step-by-step explanation or example configuration?

Sources:

- [vllm/issues/11247](https://github.com/vllm-project/vllm/issues/11247)
- [vllm/issues/14193](https://github.com/vllm-project/vllm/issues/14193)