# I got "NCCL error" when launch LLMEngine with data parallel = 2

**URL:** <https://discuss.vllm.ai/t/i-got-nccl-error-when-launch-llmengine-with-data-parallel-2/1257>\
**Category:** General\
**Created:** [July 31, 2025, 12:43pm UTC](https://discuss.vllm.ai/t/i-got-nccl-error-when-launch-llmengine-with-data-parallel-2/1257 "2025-07-31T12:43:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![IceCapriccio](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/icecapriccio/32/558_2.png) [@IceCapriccio](https://discuss.vllm.ai/u/IceCapriccio)\
**Post date:** [July 31, 2025, 12:43pm UTC](https://discuss.vllm.ai/t/i-got-nccl-error-when-launch-llmengine-with-data-parallel-2/1257/1 "2025-07-31T12:43:09Z")

</div>

Promblem Description:

when i use `swift rollout`to launch a vllm server with data-parallel-size = 2，i got some error like:

> File “/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc\_executor.py”, line 57, in collective\_rpc  
> answer = run\_method(self.driver\_worker, method, args, kwargs)  
> File “/usr/local/lib/python3.10/dist-packages/vllm/utils/ **init**.py”, line 2736, in run\_method  
> return func(\*args, \*\*kwargs)  
> File “/usr/local/lib/python3.10/dist-packages/vllm/worker/worker\_base.py”, line 606, in init\_device  
> self.worker.init\_device() # type: ignore  
> File “/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py”, line 193, in init\_device  
> init\_worker\_distributed\_environment(self.vllm\_config, self.rank,  
> File “/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py”, line 534, in init\_worker\_distributed\_environment  
> ensure\_model\_parallel\_initialized(parallel\_config.tensor\_parallel\_size,  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel\_state.py”, line 1095, in ensure\_model\_parallel\_initialized  
> initialize\_model\_parallel(tensor\_model\_parallel\_size,  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel\_state.py”, line 1061, in initialize\_model\_parallel  
> \_DP = init\_model\_parallel\_group(group\_ranks,  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel\_state.py”, line 832, in init\_model\_parallel\_group  
> return GroupCoordinator(  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel\_state.py”, line 255, in **init**  
> self.device\_communicator = device\_comm\_cls(  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/device\_communicators/cuda\_communicator.py”, line 50, in **init**  
> self.pynccl\_comm = PyNcclCommunicator(  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/device\_communicators/pynccl.py”, line 100, in **init**  
> self.comm: ncclComm\_t = self.nccl.ncclCommInitRank(  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/device\_communicators/pynccl\_wrapper.py”, line 286, in ncclCommInitRank  
> self.NCCL\_CHECK(self.\_funcs[“ncclCommInitRank”](ctypes.byref(comm),  
> File “/usr/local/lib/python3.10/dist-packages/vllm/distributed/device\_communicators/pynccl\_wrapper.py”, line 257, in NCCL\_CHECK  
> raise RuntimeError(f"NCCL error: {error\_str}")  
> **RuntimeError: NCCL error: invalid usage (run with NCCL\_DEBUG=WARN for details)**

I read ms-swift’s [source code](https://github.com/modelscope/ms-swift/blob/main/swift/llm/infer/rollout.py#L155), then i reproduce this error with the script below：

```auto
from multiprocessing import Pipe, Process
from vllm import SamplingParams, EngineArgs, LLMEngine
import os

DP = 2
connections = []
processes = []
master_port=29503

def llm_worker(data_parallel_rank: int, master_port: int, connection) -> None:
    # Set required environment variables for DP to work with vLLM
    os.environ['VLLM_DP_RANK'] = str(data_parallel_rank)
    os.environ['VLLM_DP_RANK_LOCAL'] = str(data_parallel_rank)
    os.environ['VLLM_DP_SIZE'] = str(DP)
    os.environ['VLLM_DP_MASTER_PORT'] = str(master_port)
    kwargs = {'model': '/path/to/qwen3_1.7b', 'dtype': 'float16', 'gpu_memory_utilization': 0.9, 'tensor_parallel_size': 1, 'pipeline_parallel_size': 1, 'max_model_len': None, 'max_num_seqs': 256, 'disable_log_stats': True, 'disable_custom_all_reduce': True, 'enforce_eager': False, 'trust_remote_code': True, 'enable_prefix_caching': False, 'distributed_executor_backend': None, 'device': 'cuda:0', 'seed': None, 'quantization': None, 'worker_extension_cls': 'trl.scripts.vllm_serve.WeightSyncWorkerExtension', 'enable_chunked_prefill': False, 'enable_sleep_mode': False}
    args = EngineArgs(**kwargs)
    engine = LLMEngine.from_engine_args(args)
    # engine = SwiftRolloutDeploy.get_infer_engine(args, template=args.get_template(None))
    

for data_parallel_rank in range(DP):
        parent_conn, child_conn = Pipe()
        process = Process(target=llm_worker, args=(data_parallel_rank, master_port, child_conn))
        process.start()
        connections.append(parent_conn)
        processes.append(process)

```

only when DP set to 2，the error above will occur; if i set DP to 1，the code will finish successfully.

Environment:

deepspeed==0.17.2

transformers==4.52.4

vllm==0.9.2

torch==2.7.0

GPU：8 \* V100  
NCCL：2.26（用的是python的nvidia-nccl-cu12==2.26.2）  
CUDA：12.6（用的是python的nvidia-cuda-cupti-cu12==12.6.80）  
系统：Ubuntu22.04

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [July 31, 2025, 12:43pm UTC](https://discuss.vllm.ai/t/i-got-nccl-error-when-launch-llmengine-with-data-parallel-2/1257/2 "2025-07-31T12:43:46Z")

</div>

你的环境（deepspeed==0.17.2、transformers==4.52.4、vllm==0.9.2、torch==2.6.0、8×V100、NCCL 2.21.5、CUDA 12.4、Ubuntu 22.04）是兼容的，但需注意V100显卡要求CUDA 12.x和PyTorch 2.6.0配套，且vLLM推荐新建独立环境以避免CUDA/PyTorch二进制不兼容问题。NCCL建议使用与PyTorch和CUDA版本一致的官方包，避免多版本冲突，且V100需compute capability 7.0及以上。[官方文档](https://docs.vllm.ai/en/stable/getting_started/installation/gpu.html)和[安装说明](https://github.com/vllm-project/vllm/blob/main/docs/getting_started/installation/gpu/cuda.inc.md)均有详细说明。

如遇NCCL相关问题，建议检查NCCL库版本、驱动、环境变量（如NCCL\_CUMEM\_ENABLE=0），并确保所有节点网络互通。若需多卡并行，vLLM支持通过`--tensor-parallel-size 8`参数实现。[多卡部署示例](https://github.com/vllm-project/vllm/issues/5704)可参考。需要更详细的配置或排查建议吗？

Sources:

- [vLLM官方安装文档](https://docs.vllm.ai/en/stable/getting_started/installation/gpu.html)
- [vLLM CUDA安装说明](https://github.com/vllm-project/vllm/blob/main/docs/getting_started/installation/gpu/cuda.inc.md)
- [多卡部署用法](https://github.com/vllm-project/vllm/issues/5704)