ROCm/vLLM issues compared to CUDA/vLLM

Hardware Support
1

Hello, I am developing an app that uses LLMs and needs vLLM to run.

To be honest I am not so deeply technical so I am using AI for support and during our testing we found out that performance our app is much worse on ROCm vs CUDA.

Here is detail finding prepared by Claude, if you have any questions please let me know.
________________________________________________________________

ISSUE 1

Title: [ROCm] block-FP8 GEMM tuning script is CUDA-gated, so ROCm users can't generate missing kernel configs

Labels to consider: rocm, performance, quantization

Summary

On ROCm/gfx942, serving a model whose block-FP8 GEMM shapes aren’t in the shipped config set logs
Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! — but benchmarks/kernels/benchmark_block_fp8_gemm.py, the in-tree tool for producing those configs, asserts current_platform.is_cuda() and refuses to run.

The result is a dead end: the warning tells the operator exactly what’s wrong, and there is no supported way to act on it.

Environment
GPU 1× AMD Instinct MI300X VF (SR-IOV virtual function), gfx942, 192 GB
Image rocm/vllm:rocm7.14.0_cdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0
vLLM v0.23.1.dev1+g9ddef7117.d20260715
Host Ubuntu 26.04, kernel 7.0.0-27-generic
Serving --tensor-parallel-size 1 --max-model-len 32768 --gpu-memory-utilization 0.90
Model Qwen/Qwen3.5-122B-A10B-FP8

Reproduce

# 1) serve a model whose shapes aren't in the shipped set
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3.5-122B-A10B-FP8 --tensor-parallel-size 1 \
  --max-model-len 32768 --gpu-memory-utilization 0.90
# -> logs the warning below, 5 times

# 2) try to generate the missing config
python benchmarks/kernels/benchmark_block_fp8_gemm.py --help
# AssertionError: Only support benchmarking w8a8 block fp8 kernel on CUDA device.

Observed

WARNING [fp8_utils.py:828] Using default W8A8 Block FP8 kernel config.
Performance might be sub-optimal! Config file not found at
  …/vllm/model_executor/layers/quantization/utils/configs/
  N=20480,K=3072,device_name=AMD_Instinct_MI300X,dtype=fp8_w8a8,block_shape=[128,128].json

Five distinct shapes, all device_name=AMD_Instinct_MI300X, dtype=fp8_w8a8, block_shape=[128,128]:

N K
17408 3072
2048 3072
20480 3072
3072 1024
3072 8192

This is a coverage gap, not a platform gap

Worth stating clearly, because it narrows the fix:

  • The image ships 41 tuned AMD_Instinct_MI300X configs, so MI300X is supported.
  • Device-name resolution is correct — the card reports AMD Instinct MI300X VF and vLLM looks up AMD_Instinct_MI300X, so the VF suffix is already handled. Not a naming bug.
  • The shipped set covers other models’ shapes (many at K=7168 / 2048 / 512). Qwen 3.5’s K=3072 / 1024 / 8192 are simply absent.

So the shapes are missing, not the support — and this will recur for every new model whose dimensions aren’t already in the set.

What would help, in order of usefulness

  1. Let benchmark_block_fp8_gemm.py run on ROCm, even with a different search space, so operators can generate configs for their own model’s shapes.
  2. Failing that, document how the existing AMD_Instinct_MI300X configs were produced, so the procedure is reproducible outside the project.
  3. Optionally ship configs for Qwen 3.5 122B-A10B’s five shapes — though (1) generalises and this does not.

Impact — deliberately not overstated

We measured the penalty at roughly 10–25%, not the ~2.9× difference we see against our
NVIDIA reference. Our reasoning: a model on a different quantization path (gpt-oss-120b, Triton MXFP4 MoE backend, zero such warnings) ran 2.18–2.48× faster on the same card, but it also activates ~5B params against Qwen’s ~10B, which alone predicts ~1.96×. The residual beyond active-parameter count is small. We are not claiming the missing configs explain the gap — just that they’re a real, fixable contributor and we currently can’t address them ourselves.

Happy to run a patch on this hardware and report back.

ISSUE 2

Title: [ROCm][gfx942] Nemotron 3 Super 120B-A12B-FP8 fails engine init with HIPBLAS_STATUS_NOT_SUPPORTED

Labels to consider: rocm, bug, quantization

Summary

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 loads fully on MI300X (gfx942) and then dies during engine initialization in a hipBLAS call. The model is recognised and all weights load, so this reads as a missing kernel path rather than a missing port.

Environment
GPU 1× AMD Instinct MI300X, gfx942, 192 GB
Image rocm/vllm CDNA build (rocm7.14.0_cdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0)
Model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8
Serving --tensor-parallel-size 1

Observed

vLLM resolves NemotronHForCausalLM, selects the fp8_e4m3 KV cache, applies the NemotronH mamba_ssm_cache_dtype=float32 fixup, and loads all 26 shards / 114.28 GiB — then:

RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling
  `HIPBLAS_STATUS_NOT_SUPPORTED`
Engine core initialization failed.

The failing call is a GEMM taking a bias — an FP8 linear whose shape hipBLAS appears to have no supported path for on gfx942.

Already tried

  • TORCH_BLAS_PREFER_HIPBLASLT=0 — the usual workaround for this error class. Identical failure.
  • BF16 instead of FP8 — not viable: ~230 GB exceeds a single MI300X’s 192 GB, so it would need TP=2.

Why it matters

This is one of the two 120B-class models we support. On NVIDIA both serve; on AMD only the other one does, so an AMD deployment currently has no choice of model.

We can re-run this on the same hardware and provide a fuller trace or test a patch — say what would be useful.

Reference — throughput context (optional, include only if asked)

Not part of either issue; kept here in case a maintainer asks what the numbers look like. Same GPU, same image, TP=1, idle machine, identical harness, tokens from usage.completion_tokens.

concurrency Qwen 3.5 122B-A10B-FP8 gpt-oss-120b (MXFP4)
1 58.2 tok/s 144.5 tok/s
8 379.5 704.0
32 1,346.5 2,531.6
64 2,246.8 4,888.9

Qwen single-stream 58.3 tok/s, TTFT 0.043 s, reproducing an independent run four days
earlier (58.1 tok/s). Our 2× H200 TP=2 figure for the same checkpoint is 169 tok/s.

Caveats that limit what these support:

  • The MI300X figure is one GPU at TP=1; the H200 figure is two GPUs at TP=2. For
    aggregate throughput that confound is decisive. For single-stream it is weaker — in our measurements single-stream tok/s is bandwidth-per-active-param bound and barely moved with GPU count (169 tok/s held across 2× and 8× H200) — so we read the single-stream gap as broadly real and per-GPU aggregate throughput as roughly comparable.
  • gpt-oss activates ~5B params vs Qwen’s ~10B; ~1.96× is predicted from that alone, against 2.18–2.48× measured. The comparison localises the issue to the block-FP8 path (gpt-oss emits no such warnings) but cannot attribute the speed difference to it.
  • Our streaming harness undercounts models that emit reasoning in a separate channel, so all figures are usage-based and no TTFT is quoted for gpt-oss.

One machine, one VF, one configuration per model. A deployment report, not a benchmark.

Not vLLM issues

Listed so they aren’t confused with the above: MI300X GPU firmware was absent from the host on 2 of 2 rented machines (amdgpu loads, /dev/kfd appears, zero usable GPUs until the blobs are installed from upstream linux-firmware), and /dev/kfd belongs to group render rather than video. Both are host/distro issues; we mention them only to make clear we are not attributing them to vLLM.

2

It seems likely that your performance issues on ROCm are due in part to missing block-FP8 GEMM kernel configs and incomplete kernel support for some models. Specifically:

  1. Block-FP8 GEMM Tuning on ROCm: The script benchmark_block_fp8_gemm.py is currently CUDA-only, so ROCm users cannot generate missing kernel configs for their hardware. This means when you serve models with shapes not covered by the shipped configs, vLLM falls back to default (non-tuned) kernels, resulting in 10–25% slower performance and warnings like “Performance might be sub-optimal!” There is no documented way to generate these configs on ROCm, and this gap will affect any new model with novel GEMM shapes. According to the issue, the only workaround would be for vLLM to support ROCm in the tuning script or to ship more configs for common models, but neither is currently available (source).

  2. Nemotron 3 Super 120B-A12B-FP8 Fails on ROCm: This model fails to initialize on MI300X with a HIPBLAS_STATUS_NOT_SUPPORTED error during an FP8 GEMM with bias. This suggests that the required kernel is not implemented in hipBLAS for this shape or operation. Usual workarounds like disabling hipBLASLt do not help, and switching to BF16 is not feasible due to memory constraints. This is a kernel support gap in ROCm for certain FP8 operations (source).

Would you like more detail on possible workarounds or how to monitor for future ROCm support improvements?

Sources:

Hint: Mention @Herald in the post for followups.

3

Regarding issue 1

I tested benchmarks/kernels/benchmark_block_fp8_gemm.py, which is CUDA-gated. The actual tuning script is benchmarks/kernels/benchmark_w8a8_block_fp8.py, which asserts current_platform.is_cuda() or current_platform.is_rocm() and runs correctly on gfx942 — I confirmed it starts and sweeps on an MI300X. Its help text documents the flow, and save_configs writes the filename the lookup expects (get_device_name() returns
AMD_Instinct_MI300X even on a VF card, so the names match).

So ROCm users can generate missing block-FP8 configs. There is no tooling gap here. I cited the wrong script and did not check whether another one existed.

To be precise about what I did and didn’t verify: I ran the tuner but stopped it partway (603 of 1,280 candidates on one shape/batch pair), so I have not produced a finished config or measured a before/after speedup. What I can report is the rate: ~1,280 candidate configs per (shape × batch size) at ~1.3/s and decelerating as block sizes grow — roughly 18-25 minutes per pair. Our model needs 5 shapes, and the default sweep covers 18 batch
sizes, which extrapolates to something like 30-40 GPU-hours on a single MI300X. That is an extrapolation from a partial run, not a completed measurement — but it may still be useful if you’re ever sizing expectations or considering a faster path.

One small suggestion remains, and it’s the thing that would have prevented this issue: the fp8_utils.py:828 warning names the missing config file but not the script that generates it. A pointer there would close the loop for anyone wanting to fix their own model’s shapes. Happy to open that separately as a docs/UX item if it’d be welcome.

Issue 2 is still valid