TORCH_DEVICE_IDENTITY = torch.ones(1).cuda() if is_hip() else None
^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/torch/cuda/ __init__.py", line 372, in _lazy_init
torch._C._cuda_init()
RuntimeError: No HIP GPUs are available
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
(base) aiscuser@node-0:~/verl$ pip list
DEPRECATION: Loading egg at /opt/conda/lib/python3.11/site-packages/setuptools-78.0.2-py3.11.egg is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330
Package Version Editable project location
--------------------------------- -------------------------- -------------------------
accelerate 1.5.2
aiohappyeyeballs 2.6.1
aiohttp 3.11.14
aiohttp-cors 0.8.0
aiosignal 1.3.2
amdsmi 25.1.0+8dc45db
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 4.9.0
archspec 0.2.5
attrs 25.3.0
autocommand 2.2.2
awscli 1.38.19
backports.tarfile 1.2.0
boltons 24.0.0
boto3 1.37.19
botocore 1.37.19
Brotli 1.1.0
cachetools 5.5.2
certifi 2025.1.31
cffi 1.17.1
charset-normalizer 3.4.1
click 8.1.8
cloudpickle 3.1.1
codetiming 1.4.0
colorama 0.4.6
colorful 0.5.6
conda 25.1.1
conda-libmamba-solver 25.3.0
conda-package-handling 2.4.0
conda_package_streaming 0.11.0
datasets 3.4.1
dill 0.3.8
diskcache 5.6.3
distlib 0.3.9
distro 1.9.0
docker-pycreds 0.4.0
docutils 0.16
einops 0.8.1
fastapi 0.115.12
filelock 3.16.1
flash_attn 2.7.3
frozendict 2.4.6
frozenlist 1.5.0
fsspec 2024.10.0
gguf 0.10.0
gitdb 4.0.12
GitPython 3.1.44
google-api-core 2.24.2
google-auth 2.38.0
googleapis-common-protos 1.69.2
grpcio 1.71.0
h11 0.14.0
h2 4.2.0
hiredis 3.1.0
hpack 4.1.0
httpcore 1.0.7
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.29.3
hydra-core 1.3.2
hyperframe 6.1.0
idna 3.10
importlib_metadata 8.6.1
inflect 7.3.1
iniconfig 2.1.0
inquirerpy 0.3.4
interegular 0.3.3
jaraco.collections 5.1.0
jaraco.context 5.3.0
jaraco.functools 4.0.1
jaraco.text 3.12.1
Jinja2 3.1.4
jiter 0.9.0
jmespath 1.0.1
jsonpatch 1.33
jsonpointer 3.0.0
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
lark 1.2.2
libmambapy 2.0.8
libnacl 2.1.0
liger_kernel 0.5.5
llvmlite 0.44.0
lm-format-enforcer 0.10.6
MarkupSafe 2.1.5
menuinst 2.2.0
mistral_common 1.5.4
more-itertools 10.3.0
mpmath 1.3.0
msgpack 1.1.0
msgspec 0.19.0
multidict 6.2.0
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.4.2
numba 0.61.0
numpy 1.26.4
omegaconf 2.3.0
openai 1.68.2
opencensus 0.11.4
opencensus-context 0.1.3
opencv-python-headless 4.11.0.86
orjson 3.10.16
outlines 0.0.46
packaging 24.2
pandas 2.2.3
partial-json-parser 0.2.1.1.post5
peft 0.15.0
pfzy 0.3.4
pillow 11.0.0
pip 25.0.1
platformdirs 4.3.6
pluggy 1.5.0
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.50
propcache 0.3.0
proto-plus 1.26.1
protobuf 5.29.4
psutil 7.0.0
py-cpuinfo 9.0.0
py-spy 0.4.0
pyairports 2.1.1
pyarrow 19.0.1
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybind11 2.13.6
pycosat 0.6.6
pycountry 24.6.1
pycparser 2.22
pydantic 2.10.6
pydantic_core 2.27.2
pylatexenc 2.10
PySocks 1.7.1
pytest 8.3.5
pytest-asyncio 0.26.0
python-dateutil 2.9.0.post0
python-dotenv 1.1.0
pytorch-triton-rocm 3.3.0+git96316ce5
pytz 2025.2
PyYAML 6.0.2
pyzmq 26.3.0
ray 2.44.0
redis 5.2.1
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rpds-py 0.23.1
rsa 4.7.2
ruamel.yaml 0.18.10
ruamel.yaml.clib 0.2.8
s3transfer 0.11.4
safetensors 0.5.3
scipy 1.15.2
sentencepiece 0.2.0
sentry-sdk 2.24.1
setproctitle 1.3.5
setuptools 75.8.2
setuptools 78.0.2
setuptools-scm 8.2.0
six 1.17.0
smart-open 7.1.0
smmap 5.0.2
sniffio 1.3.1
starlette 0.46.1
supervisor 4.2.5
sympy 1.13.3
tensorboardX 2.6.2.2
tensordict 0.6.2
tensorizer 2.9.2
tiktoken 0.9.0
tokenizers 0.21.1
tomli 2.0.1
torch 2.8.0.dev20250325+rocm6.3
torchaudio 2.6.0.dev20250325+rocm6.3
torchdata 0.11.0
torchvision 0.22.0.dev20250325+rocm6.3
tqdm 4.67.1
transformers 4.50.1
triton 3.2.0
truststore 0.10.1
typeguard 4.3.0
typing_extensions 4.12.2
tzdata 2025.2
urllib3 2.3.0
uvicorn 0.34.0
uvloop 0.21.0
verl 0.2.0.dev0 /verl
virtualenv 20.29.3
vllm 0.6.3+rocm634 /vllm
wandb 0.19.8
watchfiles 1.0.4
wcwidth 0.2.13
websockets 15.0.1
wheel 0.45.1
wrapt 1.17.2
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0
zstandard 0.23.0
(base) aiscuser@node-0:~/verl$ python
Python 3.11.11 | packaged by conda-forge | (main, Mar 3 2025, 20:43:55) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> import vllm
WARNING 03-26 03:00:44 rocm.py:13] `fork` method is not supported by ROCm. VLLM_WORKER_MULTIPROC_METHOD is overridden to `spawn` instead.
>>> torch.cuda.is_available()
True
>>> torch.ones(1).cuda()
tensor([1.], device='cuda:0')
>>>
(base) aiscuser@node-0:~/verl$
```
---
### Author:  [@XueruiSu](https://discuss.vllm.ai/u/XueruiSu)
#### Post date: [March 26, 2025, 3:45am UTC](https://discuss.vllm.ai/t/no-hip-gpus-are-available-for-verl/242/2 "2025-03-26T03:45:03Z")
Ask for help, wuwu. If you need more information, contact me with [bi2030592079@163.com](mailto:bi2030592079@163.com)
---
### Author:  [@youkaichao](https://discuss.vllm.ai/u/youkaichao)
#### Post date: [March 26, 2025, 2:32pm UTC](https://discuss.vllm.ai/t/no-hip-gpus-are-available-for-verl/242/3 "2025-03-26T14:32:49Z")
I think it might be related to [[Bugfix][V1] Avoid importing PreTrainedModel by HollowMan6 · Pull Request #15366 · vllm-project/vllm · GitHub](https://github.com/vllm-project/vllm/pull/15366)
---
### Author:  [@hiyouga](https://discuss.vllm.ai/u/hiyouga)
#### Post date: [March 31, 2025, 9:26am UTC](https://discuss.vllm.ai/t/no-hip-gpus-are-available-for-verl/242/4 "2025-03-31T09:26:43Z")
Hi Xuerui, could you please provide the full traceback of the verl about this issue? `No HIP GPUs are available`
---
### Author:  [@tjtanaa](https://discuss.vllm.ai/u/tjtanaa)
#### Post date: [April 4, 2025, 12:11pm UTC](https://discuss.vllm.ai/t/no-hip-gpus-are-available-for-verl/242/5 "2025-04-04T12:11:57Z")
@XueruiSu
If you are using bare metal, there might be some permissions are not setup correctly where your current user is not able to use PyTorch to access the AMD GPUs.
If you are able to launch a docker image, please launch a docker image and try to run the following steps to verify if it is the case where your bare metal user is not setup with correct permission.
**Example steps:**
1. Launch a docker image
```auto
#!/bin/bash
# this is rocm 6.3 python 3.12 with torch._scaled_mm rowwise feature
docker run -it \
--network=host \
--group-add=video \
--ipc=host \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
--device /dev/kfd \
--device /dev/dri \
rocm/pytorch:rocm6.3.4_ubuntu24.04_py3.12_pytorch_release_2.4.0 \
bash
```
1. Check if pytorch in python is able to detect the AMD GPUs.
```auto
import torch
print(torch.cuda.is_available())
```
**Additional information about vLLM RLHF support on ROCm:**
At the moment of creating this reply, the sleep mode on AMD has been enabled in this PR. [[Core][AMD] Migrate fully transparent sleep mode to ROCm platform by HollowMan6 · Pull Request #12695 · vllm-project/vllm](https://github.com/vllm-project/vllm/pull/12695)
There is a docker image built from this PR that you could use it as a base for your to setup to run VeRL `ghcr.io/embeddedllm/vllm-rocm:v0.8.2-18ed313-sleep-mode-rocm634_py310`.
I have validated using the following examples:
```auto
- git clone https://github.com/hiyouga/EasyR1.git
- cd EasyR1; python3 –m pip install –e .
- export ROCR_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
- export VLLM_USE_TRITON_FLASH_ATTN=0
- Login to your wandb else edit the examples/config.yaml to logger: ["console"]
- bash examples/qwen2_5_vl_7b_geo3k_grpo.sh
```