# How to run GGUF with vLLM and ROCM
**URL:** https://discuss.vllm.ai/t/how-to-run-gguf-with-vllm-and-rocm/2414
**Category:** General
**Created:** [March 1, 2026, 5:36pm UTC](https://discuss.vllm.ai/t/how-to-run-gguf-with-vllm-and-rocm/2414 "2026-03-01T17:36:59Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@AnttiK](https://discuss.vllm.ai/u/AnttiK)
#### Post date: [March 1, 2026, 5:36pm UTC](https://discuss.vllm.ai/t/how-to-run-gguf-with-vllm-and-rocm/2414/1 "2026-03-01T17:36:59Z")
Getting Value error, torch.bfloat16 is not supported for quantization method gguf. Supported dtypes: [torch.float16, torch.float32] [type=value\_error, input\_value=ArgsKwargs((), {‘model\_co…transfer\_config’: None}), input\_type=ArgsKwargs]
docker run -it --dns=192.168.49.1 --network=host --group-add=video --ipc=host --cap-add=SYS\_PTRACE --security-opt seccomp=unconfined --device /dev/kfd --device /dev/dri --shm-size=16g -e HUGGING\_FACE\_HUB\_TOKEN=“ -e VLLM\_SLEEP\_WHEN\_IDLE=1 -e ROCM\_VISIBLE\_DEVICES=0,1 -e HIP\_VISIBLE\_DEVICES=0,1 -e HSA\_OVERRIDE\_GFX\_VERSION=11.0.0 -e PYTORCH\_ROCM\_ARCH=“gfx1100” -e VLLM\_USE\_TRITON\_FLASH\_ATTN=0 -e GPU\_MAX\_HW\_QUEUES=1 -e NCCL\_DEBUG=WARN -e NCCL\_IB\_DISABLE=1 --restart unless-stopped --name vllm\_rocm\_gemma-3-27b-it-qat-q4\_0-gguf2 -v /home/ubuntu/vllm\_models:/models rocm/vllm-dev:nightly vllm serve /models/gemma-3-27b-it-Q5\_K\_M.gguf --host 0.0.0.0 --port 8000 --enforce-eager --served-model-name gemma3 --tensor-parallel-size 2 --trust-remote-code --dtype auto --kv-cache-dtype auto --max-model-len 1024 --max-num-seqs 12 --max-num-batched-tokens 4000 --gpu-memory-utilization 0.90 --swap-space 12 --disable-log-requests --disable-log-stats --max-log-len 100
---
### Author:  [@RunLLM](https://discuss.vllm.ai/u/RunLLM)
#### Post date: [March 1, 2026, 5:37pm UTC](https://discuss.vllm.ai/t/how-to-run-gguf-with-vllm-and-rocm/2414/2 "2026-03-01T17:37:18Z")
The error occurs because `torch.bfloat16` is not supported for GGUF quantized models in vLLM; only `torch.float16` and `torch.float32` are allowed. You must explicitly set `--dtype float16` or `--dtype float32` when serving GGUF models to avoid this error—`auto` may select an unsupported dtype depending on your hardware and model config.
According to [vllm.model\_executor.layers.quantization.gguf documentation](https://docs.vllm.ai/en/stable/api/vllm/vllm.model_executor.layers.quantization.gguf.html), GGUF quantization supports only float16 and float32 activation dtypes. Please update your command to include `--dtype float16` or `--dtype float32`.
Would you like more detail or a sample command?
Sources:
- [vllm.model\_executor.layers.quantization.gguf — vLLM](https://docs.vllm.ai/en/stable/api/vllm/vllm.model_executor.layers.quantization.gguf.html)
Hint: Mention @RunLLM in the post for followups.
---
### Author:  [@AnttiK](https://discuss.vllm.ai/u/AnttiK)
#### Post date: [March 1, 2026, 5:40pm UTC](https://discuss.vllm.ai/t/how-to-run-gguf-with-vllm-and-rocm/2414/3 "2026-03-01T17:40:00Z")
Value error, The model type ‘gemma3\_text’ does not support float16. Reason: Numerical instability. Please use bfloat16 or float32 instead. [type=value\_error, input\_value=ArgsKwargs((), {‘model’: …rocessor\_plugin’: None}), input\_type=ArgsKwargs]
@RunLLM
---
### Author:  [@RunLLM](https://discuss.vllm.ai/u/RunLLM)
#### Post date: [March 1, 2026, 5:40pm UTC](https://discuss.vllm.ai/t/how-to-run-gguf-with-vllm-and-rocm/2414/4 "2026-03-01T17:40:21Z")
The error means the model type `gemma3_text` is known to be numerically unstable with float16; you must use `bfloat16` or `float32` as the dtype. This is enforced in vLLM for Gemma3 models due to precision issues with float16, as documented in the code and issues ([see \_FLOAT16\_NOT\_SUPPORTED\_MODELS](https://docs.vllm.ai/en/latest/api/vllm/config/model/)).
Set `--dtype bfloat16` or `--dtype float32` when serving or using this model to resolve the error.
Would you like more detail or example commands?
Sources:
- [vllm.config.model — \_FLOAT16\_NOT\_SUPPORTED\_MODELS](https://docs.vllm.ai/en/latest/api/vllm/config/model/)
- [ec4ccf9e (github issue)](https://github.com/vllm-project/vllm/issues/15610)
- [e70bb3bf (github issue)](https://github.com/vllm-project/vllm/issues/18733)
Hint: Mention @RunLLM in the post for followups.
---
### Author:  [@AnttiK](https://discuss.vllm.ai/u/AnttiK)
#### Post date: [March 1, 2026, 5:50pm UTC](https://discuss.vllm.ai/t/how-to-run-gguf-with-vllm-and-rocm/2414/5 "2026-03-01T17:50:13Z")
So it looks like its not possible to run GGUF models with ROCM and 7900 XTX.