Hi,
I have passed trough 2 7900 XTX to ubuntu VM from proxmox host. I run there the vllm rocm openai nightly docker. When running the container it throws:
[2026-06-27 19:10:11] ubuntuAIhome01:727:727 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:218 NCCL WARN Missing “iommu=pt” from kernel command line which can lead to system instablity or hang!
(Worker pid=727) INFO 06-27 19:10:11 [pynccl.py:113] vLLM is using nccl==2.27.7
[2026-06-27 19:10:11] ubuntuAIhome01:727:727 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:218 NCCL WARN Missing “iommu=pt” from kernel command line which can lead to system instablity or hang!
RCCL version : 2.27.7-HEAD:96a25b5
HIP version : 7.2.53211-c2d9476115
ROCm version : 7.2.3.0-90-c2d9476115
Hostname : ubuntuAIhome01
Librccl path : /opt/rocm/lib/librccl.so.1
[2026-06-27 19:10:11] ubuntuAIhome01:728:728 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/ubuntu-22.04/22.04/build/rccl/hipify/src/init.cc:218 NCCL WARN Missing “iommu=pt” from kernel command line which can lead to system instablity or hang!
ubuntuAIhome01:727:727 [0] [FATAL ERROR]: HIP failure: ‘the operation cannot be performed in the present state’
ubuntuAIhome01:728:728 [1] [FATAL ERROR]: HIP failure: ‘the operation cannot be performed in the present state’
(Worker pid=727) ERROR 06-27 19:10:25 [multiproc_executor.py:888] WorkerProc failed to start.
(Worker pid=728) ERROR 06-27 19:10:25 [multiproc_executor.py:888] WorkerProc failed to start.
(Worker pid=727) ERROR 06-27 19:10:25 [multiproc_executor.py:888] Traceback (most recent call last):
(Worker pid=728) ERROR 06-27 19:10:25 [multiproc_executor.py:888] Traceback (most recent call last):
(Worker pid=727) ERROR 06-27 19:10:25 [multiproc_executor.py:888] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py”, line 855, in worker_main
(Worker pid=728) ERROR 06-27 19:10:25 [multiproc_executor.py:888] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py”, line 855, in worker_main
Your error is caused by missing the kernel parameter iommu=pt on your Ubuntu VM, which is required for stable multi-GPU ROCm/RCCL operation. Without it, you get system instability or hangs, and HIP operations fail with “the operation cannot be performed in the present state” (NCCL WARN Missing “iommu=pt”, NCCL/RCCL ROCm startup issues, ROCm multi-GPU issues).
To fix, add iommu=pt to your VM’s kernel command line (e.g., via GRUB), then reboot. This is required for AMD GPUs with ROCm in passthrough/virtualized environments to enable proper peer-to-peer and device memory access. See here for details and here.
Would you like step-by-step instructions for adding iommu=pt to your kernel parameters?
Yes, the kernel parameter iommu=pt must be set on the host, not inside the guest VM. This is required so that PCI passthrough (and thus ROCm/RCCL multi-GPU) works correctly for GPUs assigned to VMs. Setting it only in the guest has no effect; the host controls IOMMU behavior (NCCL WARN Missing “iommu=pt”, ROCm multi-GPU issues).
Would you like detailed steps for setting iommu=pt on your Proxmox host?
If you have already set iommu=pt on your Proxmox host and are still seeing the “NCCL WARN Missing ‘iommu=pt’” warning inside your Ubuntu VM, it may indicate that the VM is not inheriting the host’s IOMMU setting or that passthrough is not fully configured. This warning should disappear if the host is correctly set up and the VM is using PCIe passthrough for the GPUs (NCCL WARN Missing “iommu=pt”, ROCm multi-GPU issues).
Please double-check:
The host’s kernel boot parameters include iommu=pt and are active (check with cat /proc/cmdline on the host).
The VM is configured for PCIe passthrough and the GPUs are visible in the VM (check with lspci | grep VGA or rocminfo).
No additional IOMMU or ACS/virtualization settings in the BIOS are interfering.
Would you like more detailed troubleshooting steps to verify passthrough and IOMMU status?
All works when I use only 1 gpu. When I put TP=2 it does not work. I used to have this setup on bare metal and all worked. Now in Proxmox VM can get only 1 GPU working when just changing the -e HIP_VISIBLE_DEVICES=1 and tp=1 instead -e HIP_VISIBLE_DEVICES=0,1 and tp=2 ?
so something breaks when having 2 gpus ?
This is a known issue: multi-GPU (TP=2) with ROCm in a VM often fails even if single-GPU works, especially with AMD GPUs and PCI passthrough. The most common causes are incomplete IOMMU/ACS passthrough, missing or misconfigured host kernel parameters, or limitations in the VM’s PCIe topology (NCCL WARN Missing “iommu=pt”, ROCm multi-GPU issues, ROCm/Proxmox passthrough issues).
On bare metal, ROCm can access all GPUs directly, but in a VM, correct PCIe passthrough and IOMMU/ACS settings are required for multi-GPU NCCL/RCCL to work. If only one GPU works, it’s likely the VM does not have true peer-to-peer PCIe access between both GPUs, or the ROCm stack cannot see both as fully native devices.
Would you like a checklist or step-by-step guide to debug multi-GPU passthrough in Proxmox?
Setting -e NCCL_P2P_DISABLE=1 can help bypass issues with GPU peer-to-peer (P2P) communication in virtualized or PCIe-only environments, and is a common workaround for multi-GPU hangs or deadlocks in vLLM/ROCm VMs (NCCL_P2P_DISABLE=1 workaround, ROCm multi-GPU issues). HSA_ENABLE_SDMA=0 is not mentioned as a required fix in the vLLM or ROCm troubleshooting docs for this class of error.
So, try launching your container with -e NCCL_P2P_DISABLE=1. This disables P2P and often allows multi-GPU (TP=2) to work in VMs, though with some performance loss. If it still fails, further VM passthrough or IOMMU/ACS configuration may be needed.
Would you like a full example docker run command and a checklist for VM passthrough settings?
In your server BIOS, IOMMU (sometimes called VT-d for Intel or AMD-Vi for AMD) should be enabled to allow PCI passthrough to VMs. However, ACS (Access Control Services) should be disabled if you experience hangs or P2P issues, as ACS can interfere with direct GPU-to-GPU communication in virtualized environments (NCCL/IOMMU/ACS troubleshooting, NCCL hangs and ACS/IOMMU).
Summary:
IOMMU/VT-d/AMD-Vi: Enabled
ACS: Disabled
Would you like a checklist for verifying these settings and confirming GPU passthrough in your VM?
It could be possible that if your BIOS does not have an ACS (Access Control Services) option, you can ignore it—many consumer and workstation boards do not expose ACS settings, and IOMMU being enabled is the main requirement for PCI passthrough (NCCL/IOMMU/ACS troubleshooting, NCCL hangs and ACS/IOMMU). Advanced Error Reporting (AER) is not mentioned as relevant in the vLLM or ROCm troubleshooting documentation for multi-GPU passthrough or NCCL/RCCL issues.
Would you like more details on confirming your VM’s PCIe topology or further steps to debug multi-GPU ROCm in your setup?