Support for Nvidia A2 Tensor Core GPU, stuck on shm_broadcast, unsure how to debug

Hardware Support NVIDIA GPU Support
1

Hi,

Recently I am testing out running vLLM on multiple Nvidia A2 Tensor Core GPUs, but most of the models that I tested with are stuck at shm_broadcast.py, No available shared memory broadcast block found in 60 seconds. Some models that I have tested and it works across multiple Nvidia A2 Tensor Core GPUs with no issues:
Qwen/Qwen3-VL-32B-Instruct-FP8
openai/gpt-oss-120b
Qwen/Qwen3-VL-30B-A3B-Instruct

Models that stuck at shm_broadcast.py:
Qwen/Qwen3-Coder-Next-FP8
Qwen/Qwen3.6-27B-FP8
Qwen/Qwen3.6-27B
google/gemma-4-E4B-it

below are the partial logs from running google/gemma-4-E4B-it:

Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.49s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01<00:00, 1.49s/it]
(Worker_TP0 pid=137)
(Worker_TP0 pid=137) INFO 07-19 09:01:48 [default_loader.py:430] Loading weights took 1.60 seconds
(Worker_TP0 pid=137) INFO 07-19 09:01:48 [model_runner.py:302] Model loading took 8.29 GiB and 3.000243 seconds
(Worker_TP1 pid=138) INFO 07-19 09:01:48 [model_runner.py:302] Model loading took 8.29 GiB and 3.005407 seconds
(Worker_TP0 pid=137) INFO 07-19 09:01:48 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=137) INFO 07-19 09:01:58 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/3140cf42bb/rank_0_0/backbone for vLLM’s torch.compile
(Worker_TP0 pid=137) INFO 07-19 09:01:58 [backends.py:1148] Dynamo bytecode transform time: 9.35 s
(Worker_TP1 pid=138) [rank1]:W0719 09:02:00.800000 138 torch/_inductor/utils.py:1731] Not enough SMs to use max_autotune_gemm mode
(Worker_TP0 pid=137) [rank0]:W0719 09:02:00.812000 137 torch/_inductor/utils.py:1731] Not enough SMs to use max_autotune_gemm mode
(Worker_TP0 pid=137) INFO 07-19 09:02:06 [backends.py:378] Cache the graph of compile range (1, 3248) for later use
(Worker_TP0 pid=137) INFO 07-19 09:02:28 [backends.py:393] Compiling a graph for compile range (1, 3248) takes 29.36 s
(Worker_TP0 pid=137) INFO 07-19 09:02:32 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/dc237f55f9dab127e27354dcab22790840f075f2affca70411da18d19d092ea8/rank_0_0/model
(Worker_TP0 pid=137) INFO 07-19 09:02:32 [monitor.py:53] torch.compile took 43.57 s in total
(Worker_TP0 pid=137) INFO 07-19 09:02:34 [monitor.py:81] Initial profiling/warmup run took 1.71 s
(Worker_TP0 pid=137) INFO 07-19 09:02:36 [gpu_worker.py:538] Available KV cache memory: 4.91 GiB
(EngineCore pid=110) INFO 07-19 09:02:36 [kv_cache_utils.py:2146] GPU KV cache size: 407,080 tokens
(EngineCore pid=110) INFO 07-19 09:02:36 [kv_cache_utils.py:2147] Maximum concurrency for 16,384 tokens per request: 24.85x
(Worker_TP1 pid=138) INFO 07-19 09:02:36 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested.
(Worker_TP0 pid=137) INFO 07-19 09:02:36 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested.
Capturing CUDA graphs (PIECEWISE): 100%|██████████| 4/4 [00:00<00:00, 7.38it/s]
(Worker_TP1 pid=138) INFO 07-19 09:02:40 [model_runner.py:722] Graph capturing finished in 4 secs, took 0.11 GiB
Capturing CUDA graphs (FULL): 100%|██████████| 3/3 [00:03<00:00, 1.02s/it]
(Worker_TP0 pid=137) INFO 07-19 09:02:40 [model_runner.py:722] Graph capturing finished in 4 secs, took 0.11 GiB
(EngineCore pid=110) INFO 07-19 09:03:37 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=110) INFO 07-19 09:04:37 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=110) INFO 07-19 09:05:37 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=110) INFO 07-19 09:06:37 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=110) INFO 07-19 09:07:37 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=110) INFO 07-19 09:08:37 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).

Is there anything I can do to pinpoint the error/issues? For info, I am running vllm 0.25.1 docker on debian 13.5. Thank you in advance for any insights!

2

The “No available shared memory broadcast block found in 60 seconds” message means vLLM is stuck, often due to a deadlock or hang in distributed communication, not just a shared memory issue. This is commonly seen when some models work but others hang, especially on multi-GPU setups with A2 GPUs. The root cause is usually a problem with GPU peer-to-peer (P2P) communication, NCCL, or CUDA driver compatibility, particularly on lower-end GPUs or with certain models. Disabling custom all-reduce (--disable-custom-all-reduce) and setting NCCL_P2P_DISABLE=1 can sometimes bypass the issue, but not always. Increasing Docker shared memory (--shm-size) may help, but if the problem persists, it is likely a hardware/driver or NCCL compatibility issue. For more debugging, enable detailed logging with export VLLM_LOGGING_LEVEL=DEBUG and export NCCL_DEBUG=TRACE as described in the troubleshooting guide.

If these steps do not resolve the issue, try running the NCCL/GPU communication test script from the troubleshooting documentation to check for hardware or driver problems. If the test script hangs, the issue is likely outside vLLM and related to your system’s NCCL or CUDA setup. Would you like more detailed step-by-step debugging instructions?

Sources:

Hint: Mention @Herald in the post for followups.

3

Hi @Herald, thank you for the response.

I tested with the disabling custom all reduce, but there is still the same issue. Instead of shared memory, I have enabled --ipc=host instead. I have also tried running the NCCL/GPU communication test script, and it shown that the communication test passed. I added the debug environment variables, and on top of the usual No available shared memory broadcast message, there is a Waiting for 1 local, 0 remote core engine proc(s) to start Debug message from the API Server that is showing. Before that there are Poller received notify event Messages, but after that is all Poller timed out. Do let me know if you would like a copy of the logs and I can find somewhere to upload them.

One pattern we noticed were that this only affect newer models (gemma 4, qwen 3.6), as we can run the other models (gpt-oss, qwen3-VL) with no issues.

Any other debugging steps would be greatly appreciated!