It could be possible that, on dual consumer GPUs (like RTX 5090) with working BAR1 P2P and TP=2, enabling NCCL P2P (CUMEM) over SHM/direct yields only a small throughput difference—typically in the +2–5% range, sometimes negative, and highly workload/config dependent. This aligns with your findings and is echoed in prior vLLM discussions, where users note that PCIe P2P is not always faster than SHM, especially for small message sizes or certain topologies. Larger, consistent gains (10–30%) are usually seen only with additional optimizations (e.g., MoE fuse tuning, custom all-reduce) or on different hardware stacks, not from a pure NCCL P2P toggle on dense models alone. See the discussion in vllm-project/vllm#14196 and discuss.vllm.ai/t/what-means-there-is-no-p2p-support/1928.
Though I’m not entirely certain, it might be that the message-size crossover where PCIe P2P beats SHM for vLLM’s all-reduce is not explicitly documented in the data sources. However, vllm-project/vllm#36481 and vllm-project/vllm#39040 suggest that for TP=2 on PCIe, P2P can be slightly better, but the benefit is modest and may depend on topology (e.g., single PCIe switch vs PHB). For diagnostics, enabling NCCL_DEBUG=INFO or TRACE and reviewing NCCL logs is the recommended approach to confirm transport and measure collective times, as described in vllm-project/vllm#6610 and vllm-project/vllm#4277.
Would you like a more detailed breakdown of the diagnostics or message-size crossover evidence?
Sources:
- vllm-project/vllm#14196
- discuss.vllm.ai/t/what-means-there-is-no-p2p-support/1928
- vllm-project/vllm#36481
- vllm-project/vllm#39040
- vllm-project/vllm#6610
- vllm-project/vllm#4277
Hint: Mention @Herald in the post for followups.