Vllm omni and 2x 5090

[AnttiK]

Hi
Trying to run vllm omni the lastest docker image with 2x5090 and getting these:
AssertionError: DP adjusted local rank 1 is out of bounds for 1 devices.
(StageEngineCoreProc_stage0_replica0 pid=184) INFO 07-26 21:07:11 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2

docker run --runtime nvidia --gpus all
-v ~/.cache/huggingface:/root/.cache/huggingface
-e HUGGING_FACE_HUB_TOKEN=“hM”
-e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
-p 8091:8091
–ulimit memlock=-1
–ulimit stack=67108864
–ipc=host
–network host
vllm/vllm-omni:latest
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --tensor-parallel-size 2 --port 8091

[RunLLM]

The error AssertionError: DP adjusted local rank 1 is out of bounds for 1 devices occurs because the default stage config for Qwen3-Omni expects 2 GPUs, but when you pass --tensor-parallel-size 2 on the CLI, it is merged into all stages, including those that only expose a single device. This causes vLLM-Omni to try to launch a second TP worker on a non-visible device, resulting in the invalid device ordinal error. To fix this, do not pass --tensor-parallel-size 2 on the CLI; instead, use the default YAML stage config and adjust it if needed. See the discussion in vllm-omni #236 and vllm-omni #5003.

Would you like more detail on how to customize the stage config for your setup?

Sources:

• vllm-omni #236
• vllm-omni #5003

Hint: Mention @Herald in the post for followups.

[AnttiK]

@RunLLM

instead, use the default YAML stage config and adjust it if needed.

@Herald, what do you mean by that? What kind of docker run command I need then. That was in the official vllm omni docs, why it is wrong?

[RunLLM]

The official vLLM-Omni Docker docs show the correct pattern: you should not pass --tensor-parallel-size 2 for Qwen3-Omni, as the default stage config already sets up the correct parallelism for 2 GPUs. Instead, use the following command (note the absence of --tensor-parallel-size):

docker run --runtime nvidia --gpus 2 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGING_FACE_HUB_TOKEN="hM" \
  -p 8091:8091 \
  --ulimit memlock=-1 --ulimit stack=67108864 --ipc=host --network host \
  vllm/vllm-omni:latest \
  vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

This approach is documented in the official installation guide and cuda.inc.md. Passing --tensor-parallel-size on the CLI can override and break the default config, which is why it should be omitted unless you are customizing the YAML yourself.

Would you like step-by-step instructions for customizing the YAML if you need a different GPU setup?

Sources:

• Official vLLM-Omni installation guide
• cuda.inc.md

Hint: Mention @Herald in the post for followups.

[AnttiK]

@Herald, yes, I need setup for 2x5090. Without the tensor-parallel = 2 it gives out of memory. So how can I utilize both GPUs? what is the exact docker run command with 2 5090 with 32GB memory. Also I could try some other omni supported models if there is one, which would fit better on 2x5090?

[RunLLM]

To use both 2×5090 GPUs (32GB each) for Qwen3-Omni-30B-A3B-Instruct, you must edit the stage config YAML to set tensor_parallel_size: 2 for the Thinker stage, since the default config may not fit on 32GB cards. The official Docker command alone is not enough; you need to mount your custom YAML and point to it with --stage-configs-path. See this example YAML and discussion.

Example command (assuming you saved your edited YAML as /path/to/qwen3_omni_2gpu.yaml):

docker run --runtime nvidia --gpus 2 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/qwen3_omni_2gpu.yaml:/stage.yaml \
  -e HUGGING_FACE_HUB_TOKEN="hM" \
  -p 8091:8091 \
  --ulimit memlock=-1 --ulimit stack=67108864 --ipc=host --network host \
  vllm/vllm-omni:latest \
  vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --stage-configs-path /stage.yaml

For models that fit better on 2×32GB, try smaller ones like Qwen3-Omni-7B or Qwen3-Omni-14B, or check supported models.

Would you like a step-by-step YAML editing guide for your setup?

Sources:

• vllm-omni #1906

Hint: Mention @Herald in the post for followups.

[AnttiK]

@RunLLM

Would you like a step-by-step YAML editing guide for your setup?

@Herald yes, I do not know where is the yaml or should I make new one?
instruct step by step

[RunLLM]

To customize vLLM-Omni for 2×5090 GPUs, you should create a new YAML stage config file based on the default, then mount it into your Docker container. Here’s a step-by-step guide:

1. Download the default YAML:
Find the default config for Qwen3-Omni at qwen3_omni_moe.yaml. Download this file to your local machine.

2. Edit the YAML:
Open the file and set tensor_parallel_size: 2 for the Thinker stage (usually stage_id: 0). Example:

stage_args:
  - stage_id: 0
    stage_type: thinker
    runtime:
      devices: "0,1"
      tensor_parallel_size: 2
      gpu_memory_utilization: 0.9
    # ... other settings ...
  # ... other stages ...

3. Save as a new file:
Save your edited file as /path/to/qwen3_omni_2gpu.yaml.

4. Mount and use in Docker:
Run Docker with the YAML mounted and referenced:

docker run --runtime nvidia --gpus 2 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/qwen3_omni_2gpu.yaml:/stage.yaml \
  -e HUGGING_FACE_HUB_TOKEN="hM" \
  -p 8091:8091 \
  --ulimit memlock=-1 --ulimit stack=67108864 --ipc=host --network host \
  vllm/vllm-omni:latest \
  vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --stage-configs-path /stage.yaml

Let me know if you want a sample YAML or further details.

Sources:

• https://github.com/vllm-project/vllm-omni/pull/1115
• https://github.com/vllm-project/vllm-omni/issues/2139

Hint: Mention @Herald in the post for followups.

[AnttiK]

@RunLLM

--stage-configs-path /stage.yaml

@Herald what about stage 1 ? should there also be devices 0,1 and tensor parallel= 2?
Having it like this but still getting OOM out of memory:

async_chunk: true

connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
extra:
initial_codec_chunk_frames: 4
codec_chunk_frames: 25
codec_left_context_frames: 25

stages:

• stage_id: 0
  max_num_batched_tokens: 32768
  max_num_seqs: 4
  gpu_memory_utilization: 0.55
  trust_remote_code: true
  enable_prefix_caching: false
  devices: “0,1”
  tensor_parallel_size: 2
  default_sampling_params:
  temperature: 0.0
  max_tokens: 2048

• stage_id: 1
  max_num_batched_tokens: 32768
  max_num_seqs: 4
  gpu_memory_utilization: 0.4
  trust_remote_code: true
  enable_prefix_caching: false
  devices: “0,1”
  input_connectors:
  from_stage_0: connector_of_shared_memory
  default_sampling_params:
  temperature: 0.9
  top_k: 50
  max_tokens: 4096
  repetition_penalty: 1.05

• stage_id: 2
  max_num_batched_tokens: 65536
  max_num_seqs: 14
  gpu_memory_utilization: 0.1
  enforce_eager: false
  trust_remote_code: true
  enable_prefix_caching: false
  enable_chunked_prefill: false
  async_scheduling: false
  devices: “1”
  input_connectors:
  from_stage_1: connector_of_shared_memory
  default_sampling_params:
  temperature: 0.0
  top_p: 1.0
  top_k: -1
  max_tokens: 65536
  repetition_penalty: 1.1

platforms:
npu:
stages:

• stage_id: 0
  gpu_memory_utilization: 0.6
  tensor_parallel_size: 2
  max_num_batched_tokens: 8192
  compilation_config:
  cudagraph_mode: PIECEWISE # Qwen3-Omni not FULL-cudagraph-safe on NPU
  devices: “0,1”
• stage_id: 1
  gpu_memory_utilization: 0.6
  max_num_batched_tokens: 8192
  compilation_config:
  cudagraph_mode: PIECEWISE # code_predictor MaskedSelect / inner NPU graph replay
  devices: “2”
• stage_id: 2
  gpu_memory_utilization: 0.3
  compilation_config:
  cudagraph_mode: PIECEWISE
  devices: “2”

Code2Wav decode uses UpSample1d, which calls F.conv_transpose1d.

On ROCm this routes through MIOpen and is not capture-safe during

vLLM’s outer graph capture, so stage 2 runs eager to avoid MIOpen/HIP

graph-capture failures.

rocm:
stages:

• stage_id: 2
  enforce_eager: true

xpu:
stages:

• stage_id: 0
  tensor_parallel_size: 2
  enforce_eager: true
  max_cudagraph_capture_size: 0
  devices: “0,1”
• stage_id: 1
  enforce_eager: true
  max_cudagraph_capture_size: 0
  devices: “2”
• stage_id: 2
  gpu_memory_utilization: 0.7
  max_cudagraph_capture_size: 0
  devices: “2”

[RunLLM]

For Qwen3-Omni, only the Thinker stage (stage_id: 0) should use tensor_parallel_size: 2 and devices: “0,1”. The Talker (stage_id: 1) and Code2Wav (stage_id: 2) stages are much smaller and should run on a single GPU (e.g., devices: “1” for both), and should NOT use tensor_parallel_size: 2. Setting tensor_parallel_size: 2 and devices: “0,1” for stage 1 will cause unnecessary memory usage and likely OOM on 32GB cards (see discussion, official YAML example).

Recommended config for 2x32GB:

• stage 0: devices: “0,1”, tensor_parallel_size: 2, gpu_memory_utilization: 0.55
• stage 1: devices: “1”, tensor_parallel_size: 1 (omit), gpu_memory_utilization: 0.4
• stage 2: devices: “1”, tensor_parallel_size: 1 (omit), gpu_memory_utilization: 0.1

Example YAML snippet:

stages:
  - stage_id: 0
    devices: "0,1"
    tensor_parallel_size: 2
    gpu_memory_utilization: 0.55
    # ...other args...
  - stage_id: 1
    devices: "1"
    gpu_memory_utilization: 0.4
    # ...other args...
  - stage_id: 2
    devices: "1"
    gpu_memory_utilization: 0.1
    # ...other args...

This layout is verified to work on 2x48GB and should fit on 2x32GB with reduced batch size or max_num_batched_tokens if needed (see here). If you still get OOM, try lowering max_num_batched_tokens or gpu_memory_utilization further.

Would you like a full sample YAML for copy-paste?

Sources:

• vllm-omni #1906

Hint: Mention @Herald in the post for followups.