# vLLM does not work with 2x 5090 in tp 2 **URL:** \ **Category:** General\ **Created:** [September 18, 2025, 4:43pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630 "2025-09-18T16:43:08Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 18, 2025, 4:43pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/1 "2025-09-18T16:43:08Z")
Hi, I cant get 2x 5090 work with vLLM (latest docker) in tp = 2. I can see that there is some patch: > > > Greetings to all, > > I’m trying to run various LLM inference engines (vLLM, Tensor…RT-LLM) on a machine with two RTX 5090 GPUs, specifically using tensor parallelism with a size of 2. Inference on a single GPU works fine, but I’m having trouble with two GPUs. Forcing NCCL\_P2P\_DISABLE=1 did not help to resolve the issue. > > Could someone confirm whether NCCL P2P communication is supported by the RTX 5090? > > Thanks in advance! > > My setup: > \`\`\` > PyTorch version: 2.7.0a0+ecf3bae40a.nv25.02 > Is debug build: False > CUDA used to build PyTorch: 12.8 > ROCM used to build PyTorch: N/A > > OS: Ubuntu 24.04.1 LTS (x86\_64) > GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 > Clang version: 18.1.3 (1ubuntu1) > CMake version: version 3.31.6 > Libc version: glibc-2.39 > > Python version: 3.12.3 (main, Feb 4 2025, 14:48:35) \[GCC 13.3.0\] (64-bit runtime) > Python platform: Linux-5.14.0-503.29.1.el9\_5.x86\_64-x86\_64-with-glibc2.39 > Is CUDA available: True > CUDA runtime version: 12.8.61 > CUDA\_MODULE\_LOADING set to: LAZY > GPU models and configuration: > GPU 0: NVIDIA GeForce RTX 5090 > GPU 1: NVIDIA GeForce RTX 5090 > > Nvidia driver version: 570.124.06 > cuDNN version: Probably one of the following: > /usr/lib/x86\_64-linux-gnu/libcudnn.so.9.7.1 > /usr/lib/x86\_64-linux-gnu/libcudnn\_adv.so.9.7.1 > /usr/lib/x86\_64-linux-gnu/libcudnn\_cnn.so.9.7.1 > /usr/lib/x86\_64-linux-gnu/libcudnn\_engines\_precompiled.so.9.7.1 > /usr/lib/x86\_64-linux-gnu/libcudnn\_engines\_runtime\_compiled.so.9.7.1 > /usr/lib/x86\_64-linux-gnu/libcudnn\_graph.so.9.7.1 > /usr/lib/x86\_64-linux-gnu/libcudnn\_heuristic.so.9.7.1 > /usr/lib/x86\_64-linux-gnu/libcudnn\_ops.so.9.7.1 > HIP runtime version: N/A > MIOpen runtime version: N/A > Is XNNPACK available: True > > CPU: > Architecture: x86\_64 > CPU op-mode(s): 32-bit, 64-bit > Address sizes: 52 bits physical, 57 bits virtual > Byte Order: Little Endian > CPU(s): 48 > On-line CPU(s) list: 0-47 > Vendor ID: AuthenticAMD > Model name: AMD Ryzen Threadripper 7960X 24-Cores > CPU family: 25 > Model: 24 > Thread(s) per core: 2 > Core(s) per socket: 24 > Socket(s): 1 > Stepping: 1 > CPU(s) scaling MHz: 17% > CPU max MHz: 5665.0000 > CPU min MHz: 545.0000 > BogoMIPS: 8387.55 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr\_opt pdpe1gb rdtscp lm constant\_tsc rep\_good amd\_lbr\_v2 nopl nonstop\_tsc cpuid extd\_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4\_1 sse4\_2 movbe popcnt aes xsave avx f16c rdrand lahf\_lm cmp\_legacy svm extapic cr8\_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr\_core perfctr\_nb bpext perfctr\_llc mwaitx cpb cat\_l3 cdp\_l3 hw\_pstate ssbd mba perfmon\_v2 ibrs ibpb stibp ibrs\_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt\_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha\_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm\_llc cqm\_occup\_llc cqm\_mbm\_total cqm\_mbm\_local avx512\_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd\_ppin cppc arat npt lbrv svm\_lock nrip\_save tsc\_scale vmcb\_clean flushbyasid decodeassists pausefilter pfthreshold avic v\_vmsave\_vmload vgif x2avic v\_spec\_ctrl vnmi avx512vbmi umip pku ospke avx512\_vbmi2 gfni vaes vpclmulqdq avx512\_vnni avx512\_bitalg avx512\_vpopcntdq la57 rdpid overflow\_recov succor smca fsrm flush\_l1d debug\_swap > Virtualization: AMD-V > L1d cache: 768 KiB (24 instances) > L1i cache: 768 KiB (24 instances) > L2 cache: 24 MiB (24 instances) > L3 cache: 128 MiB (4 instances) > NUMA node(s): 1 > NUMA node0 CPU(s): 0-47 > Vulnerability Gather data sampling: Not affected > Vulnerability Itlb multihit: Not affected > Vulnerability L1tf: Not affected > Vulnerability Mds: Not affected > Vulnerability Meltdown: Not affected > Vulnerability Mmio stale data: Not affected > Vulnerability Reg file data sampling: Not affected > Vulnerability Retbleed: Not affected > Vulnerability Spec rstack overflow: Mitigation; Safe RET > Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl > Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and \_\_user pointer sanitization > Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected > Vulnerability Srbds: Not affected > Vulnerability Tsx async abort: Not affected > > Versions of relevant libraries: > \[pip3\] numpy==1.26.4 > \[pip3\] nvidia-ml-py==12.570.86 > \[pip3\] pytorch-triton==3.1.0+cf34004b8.internal > \[pip3\] pyzmq==26.2.1 > \[pip3\] torch==2.7.0a0+ecf3bae40a.nv25.2 > \[pip3\] torchvision==0.22.0a0+ecf3bae40a.nv25.2 > \[pip3\] transformers==4.49.0 > \[pip3\] triton2==3.2.0 > \[pip3\] tritonfrontend==2.55.0 > \[pip3\] tritonserver==0.0.0 > \[conda\] Could not collect > ROCM Version: Could not collect > Neuron SDK Version: N/A > vLLM Version: 0.7.4.dev254+ged6ea065.d20250311 > vLLM Build Flags: > CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled > GPU Topology: > GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID > GPU0 X NODE 0-47 0 N/A > GPU1 NODE X 0-47 0 N/A > > Legend: > > X = Self > SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) > NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node > PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) > PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) > PIX = Connection traversing at most a single PCIe bridge > NV# = Connection traversing a bonded set of # NVLinks > > NVIDIA\_VISIBLE\_DEVICES=all > CUBLAS\_VERSION=12.8.3.14 > NCCL\_P2P\_DISABLE=1 > NVIDIA\_REQUIRE\_CUDA=cuda\>=9.0 > CUDA\_CACHE\_DISABLE=1 > NCCL\_VERSION=2.25.1 > NVIDIA\_DRIVER\_CAPABILITIES=compute,utility,video > NVIDIA\_PRODUCT\_NAME=Triton Server > CUDA\_VERSION=12.8.0.038 > CUDNN\_FRONTEND\_VERSION=1.10.0 > CUDNN\_VERSION=9.7.1.26 > NVIDIA\_TRITON\_SERVER\_VERSION=25.02 > LD\_LIBRARY\_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/lib:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 > NVIDIA\_BUILD\_ID=143749533 > CUDA\_DRIVER\_VERSION=570.86.10 > NVIDIA\_REQUIRE\_JETPACK\_HOST\_MOUNTS= > VLLM\_USE\_V1=1 > TORCHINDUCTOR\_CACHE\_DIR=/tmp/torchinductor\_root > NCCL\_CUMEM\_ENABLE=0 > TORCHINDUCTOR\_COMPILE\_THREADS=1 > CUDA\_MODULE\_LOADING=LAZY > > > \`\`\` But its not applied yet? Why vLLM latest docker does not have this patch? This prevents to use vLLM with multipe 50 series cards and also 6000 series? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 18, 2025, 4:43pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/2 "2025-09-18T16:43:47Z")
NCCL P2P communication is supported on RTX 5090, but there is a known NCCL bug affecting multi-GPU (tensor parallelism \>1) setups with 50xx/60xx series cards, including the 5090, in vLLM and other inference engines. The issue is caused by older NCCL versions (e.g., nvidia-nccl-cu12==2.25.1 or 2.26.2) bundled in the official vLLM Docker images, which do not include the upstream patch that fixes P2P for these GPUs. Forcing `NCCL_P2P_DISABLE=1` does not resolve the problem in most cases. The fix is to upgrade NCCL to at least 2.26.5, 2.27.3, or newer (e.g., 2.27.6 or 2.27.7) inside your Docker/container environment. This has been confirmed to resolve the issue for 5090 and Blackwell cards in vLLM with tensor parallelism \>1, and is also required for some 6000 series cards ([source](https://github.com/vllm-project/vllm/issues/19166), [source](https://github.com/vllm-project/vllm/issues/21491), [source](https://github.com/vllm-project/vllm/issues/23253), [source](https://github.com/vllm-project/vllm/issues/23781), [source](https://github.com/vllm-project/vllm/issues/14628)). The official vLLM Docker images may not yet include the fixed NCCL version by default, so you must manually update it. For example, add `RUN pip install nvidia-nccl-cu12==2.27.7` to your Dockerfile after the base image is pulled. This is a widely reported and resolved issue, and upgrading NCCL is the recommended solution ([source](https://github.com/vllm-project/vllm/issues/19166), [source](https://github.com/vllm-project/vllm/issues/21491), [source](https://github.com/vllm-project/vllm/issues/23253), [source](https://github.com/vllm-project/vllm/issues/23781), [source](https://github.com/vllm-project/vllm/issues/14628)). Would you like step-by-step instructions for updating your Docker image? Sources: - [Multi GPU inference using two RTX 5090s(TP=2)](https://github.com/vllm-project/vllm/issues/14628) - [CUDA Illegal Access Error when running 2x Blackwell for tensor or pipeline parallel.](https://github.com/vllm-project/vllm/issues/19166) - [Tensor parallelism on sm\_120 (rtx 5090) is broken on latest docker (0.9.2)?](https://github.com/vllm-project/vllm/issues/21491) - [latest v0.10.1 image cannot run Qwen3-Coder-30B-A3B-Instruct-FP8 on 2xRTX5090](https://github.com/vllm-project/vllm/issues/23253) - [https://github.com/vllm-project/vllm/issues/23781](https://github.com/vllm-project/vllm/issues/23781) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 18, 2025, 4:48pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/3 "2025-09-18T16:48:10Z")
Can you give me step by step instructions how to upgrade the nccl to 2.26.5 ? I have pulled the latest vLLM docker. and I am currently in the VM, not in the docker. So what should I do next? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 18, 2025, 4:48pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/4 "2025-09-18T16:48:52Z")
To upgrade NCCL to 2.26.5 in your vLLM Docker container, follow these steps: 1. **Start a shell in your running vLLM Docker container** (replace `vllm_container` with your actual container name or ID): ```bash docker exec -it vllm_container /bin/bash ``` 1. **Uninstall the old NCCL version and install 2.26.5** inside the container: ```bash pip uninstall -y nvidia-nccl-cu12 pip install nvidia-nccl-cu12==2.26.5 ``` 1. **Restart the vLLM server** inside the container if it was running. This will ensure vLLM uses NCCL 2.26.5, which fixes the multi-GPU issue on RTX 5090. For more details, see [this discussion](https://github.com/vllm-project/vllm/issues/19166) and [this example Dockerfile snippet](https://github.com/vllm-project/vllm/issues/21633). Would you like a Dockerfile example for future builds? Sources: - [CUDA Illegal Access Error when running 2x Blackwell for tensor or pipeline parallel.](https://github.com/vllm-project/vllm/issues/19166) - [v0.10.0 built with early version of pytorch that does not support sm-120](https://github.com/vllm-project/vllm/issues/21633) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 18, 2025, 4:53pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/5 "2025-09-18T16:53:46Z")
I updated the package to a docker container which was run with a command: –tensor-parallel-size 1 how can I now change that 1 to 2 ? docker start wont to do it. I am new to docker. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 18, 2025, 4:54pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/6 "2025-09-18T16:54:06Z")
You cannot change the `--tensor-parallel-size` argument of a running container. You must stop and remove the current container, then start a new one with the desired argument. For example, stop and remove the old container: ```bash docker stop docker rm ``` Then run a new container with `--tensor-parallel-size 2`: ```bash docker run --runtime nvidia --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 8000:8000 \ --ipc=host \ vllm/vllm-openai:latest \ --model \ --tensor-parallel-size 2 ``` See [official Docker usage docs](https://docs.vllm.ai/en/latest/deployment/docker.html) for more details. Would you like more step-by-step guidance? Sources: - [official Docker usage docs](https://docs.vllm.ai/en/latest/deployment/docker.html) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 18, 2025, 4:57pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/7 "2025-09-18T16:57:49Z")
> [@youlearnit](#): > > I updated the package to a docker container which was run with a command: > –tensor-parallel-size 1 > > how can I now change that 1 to 2 ? > docker start wont to do it. I am new to docker. But then I have the old nccl again in it which I just updated to a running container. So how can I update the nccl in the vllm docker and then run it with tensor paralle = 2 ? Give me step by step instructions, where the goal is to update the current vLLM docker container to work with multiple 5090 (update nccl package) and then being able to run it with the update package ---
**Author:** ![Phaelon74](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/phaelon74/32/729_2.png) [@Phaelon74](https://discuss.vllm.ai/u/Phaelon74)\ **Post date:** [September 18, 2025, 6:39pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/8 "2025-09-18T18:39:05Z")
Take the docker launch command above, feed that to ChatGPT, Gemini, whatever you use and then also comment about NCCL upgrade, and then ask it to create a Docker-Compose.yml file for you. Attempting to do that NCCL upgrade via run command is a pain. If you don’t want to do that, then everytime you launch the container with the updated TP command, you need to Exec into it and execute the NCCL upgrade command. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 18, 2025, 6:39pm UTC](https://discuss.vllm.ai/t/vllm-does-not-work-with-2x-5090-in-tp-2/1630/9 "2025-09-18T18:39:39Z")
You’re correct: upgrading NCCL via `docker exec` is only temporary—each new container will revert to the original image. The best practice is to create a custom Docker image with NCCL 2.26.5 pre-installed, then use that image in your `docker-compose.yml`. This way, every time you launch the container (with any `--tensor-parallel-size`), the correct NCCL version is present automatically. **Step 1: Create a Dockerfile to upgrade NCCL** ```dockerfile FROM vllm/vllm-openai:latest RUN pip uninstall -y nvidia-nccl-cu12 && pip install nvidia-nccl-cu12==2.26.5 ``` **Step 2: Build your custom image** ```bash docker build -t vllm-nccl-2.26.5 . ``` **Step 3: Create a `docker-compose.yml` file** ```yaml version: "3.9" services: vllm: image: vllm-nccl-2.26.5 runtime: nvidia environment: - NVIDIA_VISIBLE_DEVICES=all volumes: - ~/.cache/huggingface:/root/.cache/huggingface ports: - "8000:8000" ipc: host shm_size: 16g command: > --model --tensor-parallel-size 2 ``` Now, launching with `docker-compose up` will always use the correct NCCL version and your desired tensor parallelism. For more, see [official Docker deployment docs](https://docs.vllm.ai/en/latest/deployment/docker.html). Would you like more details or help customizing this for your setup? Sources: - [v0.10.0 built with early version of pytorch that does not support sm-120](https://github.com/vllm-project/vllm/issues/21633)