## Summary
On dual **RTX 5090** (consumer, no NVLink, PCIe **PHB** topology) we use a **patched open NVIDIA kernel module** so BAR1 GPU P2P is available (`nvidia-smi topo -p2p` **OK**, `can_device_access_peer` true). Stock GeForce software policy on this host previously reported P2P unavailable without that patch.
We ran a **controlled, single-pass** vLLM HTTP A/B where the intended change was only `NCCL_P2P_DISABLE` (`1` vs `0`). NCCL logs confirmed different transports:
| Arm | `NCCL_P2P_DISABLE` | NCCL log (`NCCL_DEBUG=INFO`) |
|—|—|—|
| off | `1` | `via SHM/direct/direct` (`isAllDirectP2p 0`) |
| on | `0` | `via P2P/CUMEM` (`isAllDirectP2p 1`) |
**Result (single timed aggregate per cell):** unweighted mean of nine cellwise % changes = **+2.67%**; range **−1.9% … +4.7%**; **8/9** cells favored P2P-on. This is an exploratory point-estimate matrix, **not** a multi-rep confidence interval. We are **not** claiming P2P is always better on consumer TP=2.
On the **same machine**, an earlier **multi-rep** long coding A/B with P2P on was **−5–7%**. That result is **not** invalidated by this matrix; sign appears **workload/config-dependent**.
Custom all-reduce was **off** for both arms (`–disable-custom-all-reduce`).
-–
## Software / hardware (pinned)
| Item | Value |
|—|—|
| GPUs | 2× GeForce RTX 5090 (SM120), TP=2 |
| Topology | `nvidia-smi topo -m`: **PHB** (no NVLink) |
| Driver | Open KMD **610.43.03** (aikitoria-style BAR1 P2P enablement) |
| IOMMU | `amd_iommu=on iommu=pt` |
| vLLM | `0.1.dev493+g4d9600003` |
| PyTorch | `2.13.0+cu130` |
| CUDA (torch) | 13.0 |
| NCCL (torch) | 2.29.7 |
| Model | ThinkingCap-Qwen3.6-27B-FP8 (dense), `kv_cache_dtype=fp8` |
| Spec | Dynamic MTP/DSD `[[1,2,2],[3,16,0]]` |
| Serving | OpenAI HTTP, non-streaming; prefix caching on; `max_num_batched_tokens=8192`; `max_num_seqs=8` |
| Parsers | reasoning + tool parsers **on** (production-like) |
| Local perf patches | launcher enables MRV2 + local K=0/prefill-related env flags (same both arms) |
| Comm | NCCL only; custom AR **disabled** both arms |
| Clocks (both arms) | ~**16051 MHz** mem OC; power limit **550 W** (default PL 575 W) |
| Display note | Desktop/compositor processes were present on a 5090 during the run; peer **integrity still passed**. Prior work on this host found display-on-5090 can break P2P integrity in other conditions—treat as a risk factor, not “headless certified.” |
### Preflight
- Peer access true both ways
- Integrity copy 0↔1: **pass**
- ~16 MiB peer bandwidth: **~14.1 GB/s** (this host historically ~1.9 GB/s through-CPU)
-–
## Method
**Intended independent variable:** `NCCL_P2P_DISABLE` ∈ {1, 0}.
Service restarted between arms. Same weights, flags, prompts, concurrency, clocks.
Metric: **aggregate output tokens / wall time** (includes prefill + decode; not pure decode tok/s).
| Workload | Input | Out | Rounds | Sessions |
|—|—|—:|—:|—
| short | ~50 tok | 256 | 3 | 1, 4, 8 |
| mid | label 8k | 256 | 2 | 1, 4, 8 |
| long | label 32k (same prompt builder used elsewhere for ~49k-scale prompts) | 256 | 2 | 1, 4, 8 |
**Limitation:** one timed aggregate per cell; no CI. Differences of a few percent **may** be run-to-run noise until multi-rep confirmation.
-–
## Results (aggregate output tok/s)
| Workload | s | P2P off (SHM) | P2P on (CUMEM) | Δ% |
|—|—:|—:|—:|—
| short | 1 | 108.5 | 111.0 | +2.3% |
| short | 4 | 220.9 | 224.5 | +1.6% |
| short | 8 | 398.4 | 408.0 | +2.4% |
| mid | 1 | 58.7 | 60.9 | +3.6% |
| mid | 4 | 128.3 | 132.6 | +3.4% |
| mid | 8 | 184.8 | 181.4 | **−1.9%** |
| long | 1 | 29.2 | 30.6 | +4.7% |
| long | 4 | 109.4 | 113.1 | +3.4% |
| long | 8 | 151.2 | 158.0 | +4.5% |
| Stat | Value |
|—|—|
| Unweighted mean of 9 cellwise Δ% | **+2.67%** |
| Min / max cell Δ | −1.9% / +4.7% |
| Cells on > off | 8 / 9 |
-–
## Interpretation
1. **Transport change is real** (SHM vs P2P/CUMEM), not a silent no-op.
2. **Observed point estimates are small and mixed** (−1.9%…+4.7%); the matrix **suggests** a mild benefit under this stack but **does not establish** it.
3. **Hypothesis only:** longer/prefill-heavier cells may benefit more from bandwidth; this run did **not** measure collective sizes or split prefill vs decode time.
4. **Prior multi-rep coding A/B (−5–7% with P2P on)** used different workload/config (e.g. MTP=3 coding harness). Configurations differed; **this single-pass matrix does not invalidate that result**.
5. Public larger “+10–30%” reports (e.g. dual 3090 + driver patch + MoE fuse tuning + other changes) are **compound experiments**, not pure NCCL toggles on dense 27B FP8.
-–
## Questions
1. On dual consumer GPUs with **working** BAR1 P2P, what **dense TP=2** tok/s deltas do you see for NCCL **SHM vs P2P/CUMEM**, with transport confirmed in logs?
2. Has anyone measured the **message-size crossover** for vLLM’s all-reduce pattern (many small ops/decode step) where PCIe P2P beats SHM?
3. For **CUDA 13.0 / NCCL 2.29.x / SM120**, which **supported diagnostics** best attribute time to NCCL collectives or identify SHM/P2P crossover (prefer measurements over knob laundry lists)?
-–
## Related (context only)
- Multi-4090 discussion (participant note: P2P not default; PCIe P2P may be slow): vllm can not start normally with multi 4090 gpus · vllm-project/vllm · Discussion #14196 · GitHub
- Dual 5090 Proxmox VM + custom-allreduce/P2P warning (not a patched-P2P perf A/B): What means there is no P2P support
- Compound patched-P2P + MoE tuning write-up (different HW/stack):
Happy to share exact serve flags or NCCL log excerpts if useful.