Most prefill–decode disaggregation numbers assume RDMA between the nodes. I wanted to know what happens on a few single-GPU cloud instances connected over plain TCP, so I measured it. The direction will not surprise anyone who has tuned chunked prefill; the size of the effects and the exceptions are what I wanted to measure.
Short version: tune the chunk first. Over TCP, disaggregation did not beat replicas with a tuned chunk at 4K inputs or in chat. A model calibrated on the runs says it pays off only for long inputs, roughly 14K–37K tokens.
- Instances: AWS g7e, g6e, g7 2xlarge, one GPU each, no EFA, so NIXL/UCX moves the KV cache over TCP
- Software and models: stock vLLM 0.30.0 with NIXL 1.4.1; Gemma 4 26B-A4B (NVFP4, FP8) and 31B (NVFP4)
- Runs: 455 over nine rounds. Each request is judged by its own TTFT (≤ 1 s) and longest inter-token gap (≤ 50 ms); a load is supported when 90% of requests pass
The four results behind that, three measured and one modeled:
| Result | |
|---|---|
| Replica stall | The prefill chunk sets the replicas’ longest gap. 26B at 4K, median 48 ms at --max-num-batched-tokens 2560, 21 ms at 512. System-wide ITL p99 does not show it. |
| Head to head | Tuned replicas were never beaten. Against replicas with a tuned chunk (26B 512–1,024, 31B 256), 1P1D over TCP served more under none of the 192 TTFT × gap-limit cells I could rank, at 4K inputs and in ShareGPT chat. Its ~3× win existed only against replicas held at 2,560 tokens. |
| Transfer | The transfer is the cost, and UCX’s default is the first fix. Most of 1P1D’s extra TTFT was the KV transfer. UCX_TCP_TX_SEG_SIZE=256k UCX_TCP_RX_SEG_SIZE=256k alone gave 3.4× (4.45 → 14.97 Gbps at low load); after that the per-flow cap (4.97 Gbps, 9.54 in a cluster placement group) and NIXL’s four-way split set the rate. |
| Where chunking runs out | Long inputs are where disaggregation pays. A step-time model calibrated on the runs predicts that 1P1D over TCP pays off at roughly 14K–37K input tokens (50 ms gap limit, low load, TTFT limit ≥ 1 s, small KV per token). Prediction, not measurement. |
Four vLLM-specific details that cost me time:
- With video enabled, Gemma 4’s multimodal budget check (
max_tokens_per_mm_item= 2,496) rejects a smaller--max-num-batched-tokens; image, audio, and video all have to be 0 in--limit-mm-per-prompt. That is exactly the case where disaggregation won. - NIXL splits a transfer into up to four pieces over four TCP connections, and the count is not exposed through
kv_connector_extra_config. With all four busy, 31B transfers aggregated 19–21 Gbps, about half of four iperf flows. - The two UCX segment sizes alone gave the 3.4×. Pinning
UCX_TCP_SNDBUF/RCVBUFat 8 MB on top was 5–10% slower (receiver’s TCP window scale ended up at 2). - The NIXL example
toy_proxy_server.pypauses all streams for 25–40 ms during gen-2 GC, and an aiohttp round-robin proxy at its default 100-connection limit silently cost the replicas one load step.
Limits: one author, one model family, three instance types, no RDMA baseline on the same hardware, and the 14K–37K boundary is a calibrated prediction with a wide band (lower edge 11K–34K). The protocol was committed before the first run; the report lists every deviation.
- Technical report (PDF): Chunk First: When Prefill–Decode Disaggregation Pays Off over TCP. Measurements and a Calibrated Model on Single-GPU Cloud Instances | Zenodo
- Code, configurations, analysis, and per-request raw data: GitHub - sunghyunjun/pd-disaggregation-over-tcp: Chunked prefill vs. prefill-decode disaggregation over TCP on single-GPU cloud instances: measurements, calibrated model, and per-request data (vLLM 0.30.0, NIXL/UCX, Gemma 4). Technical report doi:10.5281/zenodo.23085600 · GitHub (archived on Zenodo, doi:10.5281/zenodo.23085749)
Corrections welcome, especially on the NIXL/UCX details.