Install using --torch-backend=cu129 but try to import cu13

General
1

I want to serve DeepSeek-v4 w/ H100, driver version 535.129.03, so install vllm w/ uv pip install vllm --torch-backend=cu129 as described at v0.20.0 releases.

After installing, tried to serve uv run vllm serve ~~, below error occured:

Traceback (most recent call last):
  File "/user-volume/latest_serve/.venv/bin/vllm", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 18, in main
    import vllm.entrypoints.cli.benchmark.main
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/benchmark/main.py", line 10, in <module>
    from vllm.entrypoints.utils import VLLM_SUBCMD_PARSER_EPILOG
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/entrypoints/utils.py", line 19, in <module>
    from vllm.engine.arg_utils import EngineArgs
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 35, in <module>
    from vllm.config import (
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/config/__init__.py", line 6, in <module>
    from vllm.config.compilation import (
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/config/compilation.py", line 22, in <module>
    from vllm.platforms import current_platform
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/platforms/__init__.py", line 278, in __getattr__
    _current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/utils/import_utils.py", line 109, in resolve_obj_by_qualname
    module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/user-volume/latest_serve/.venv/lib/python3.12/site-packages/vllm/platforms/cuda.py", line 21, in <module>
    import vllm._C  # noqa
    ^^^^^^^^^^^^^^
ImportError: libcudart.so.13: cannot open shared object file: No such file or directory

Same error occured when using --torch-backend=cu128, etc. How can I solve it? Using latest cuda version raise below error, and can’t upgrade driver right now:

RuntimeError: The NVIDIA driver on your system is too old (found version 12020). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.
2

Your error ImportError: libcudart.so.13: cannot open shared object file: No such file or directory means the CUDA 13 runtime library is missing from your system, but vLLM was installed with CUDA 13 support. Your driver (535.129.03) only supports up to CUDA 12.2, so CUDA 13 binaries will not work. You cannot use --torch-backend=cu129 or cu130 unless your system has the matching CUDA runtime and a newer driver installed.

To fix this, install vLLM and PyTorch with CUDA 12.2 or 12.1 support, matching your driver. For example, use:

uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.19.0/cu121 --extra-index-url https://download.pytorch.org/whl/cu121 --index-strategy unsafe-best-match

If you need CUDA 12.2, replace cu121 with cu122 in the URLs. You cannot use CUDA 13 wheels or runtime until you upgrade your NVIDIA driver to a version that supports CUDA 13+ (source, source, source). Would you like more detail or step-by-step instructions?

Sources:

Hint: Mention @RunLLM in the post for followups.

4

@RunLLM

I successfully running Qwen3.5 family w/ vllm==0.19 --torch-backend cu129, so this statement is wrong: You cannot use --torch-backend=cu129 unless your system has the matching CUDA runtime and a newer driver installed.

5

You are correct that, in some cases, vLLM with --torch-backend=cu129 can run on a system with an older driver (e.g., 535.129.03) if you install the CUDA 12.9 compatibility libraries (cuda-compat-12-9) and set LD_LIBRARY_PATH accordingly. This enables forward compatibility for certain professional/datacenter GPUs, allowing CUDA 12.9 binaries to run on drivers that do not natively support CUDA 12.9 (NVIDIA CUDA compatibility, vLLM troubleshooting).

However, if you do not install the compatibility libraries, or if your GPU/driver combination does not support forward compatibility, you will encounter the ImportError for libcudart.so.13 or similar errors. So, the statement is not universally wrong, but there are supported workarounds for some environments. Would you like a step-by-step guide for setting up CUDA compatibility?

Sources:

Hint: Mention @RunLLM in the post for followups.

6

@RunLLM Install cuda-nvcc-12-9 cuda-cudart-dev-12-9 libcublas-dev-12-9 libcusparse-dev-12-9 libcusolver-dev-12-9 cuda-compat-12-9 and set LD_LIBRARY_PATH, but same error occured. Can you provide step-by-step guide for setting CUDA compatibility?

When serving Qwen3.5 family w/ vllm==0.19, cu126, just install cuda-nvcc-12-6 cuda-cudart-dev-12-6 libcublas-dev-12-6 libcusparse-dev-12-6 libcusolver-dev-12-6 works.

7

To enable CUDA compatibility for vLLM with CUDA 12.9 (cu129) on a system with an older driver, you must install the cuda-compat-12-9 package and set the environment variables as described in the official troubleshooting guide. Installing only the runtime and dev libraries is not enough—cuda-compat provides forward compatibility libraries required for this setup (vLLM troubleshooting).

Step-by-step guide:

  1. Install the compatibility package:
    sudo apt-get install cuda-compat-12-9
  2. Set environment variables before running vLLM:
    export VLLM_ENABLE_CUDA_COMPATIBILITY=1
export VLLM_CUDA_COMPATIBILITY_PATH="/usr/local/cuda-12.9/compat"
export LD_LIBRARY_PATH="/usr/local/cuda-12.9/compat:$LD_LIBRARY_PATH"
  3. Run a minimal test to verify:
    python3 -c "import vllm; import torch; print(torch.cuda.is_available())"
  4. If this works, run your vLLM serve command as usual.

Note: Forward compatibility only works on select professional/datacenter GPUs. For consumer GPUs, this method may not work (NVIDIA CUDA compatibility). Would you like more troubleshooting steps?

Sources:

Hint: Mention @RunLLM in the post for followups.

8

@RunLLM Install & setting options you mentioned, terminal test returns True, but same ImportError: libcudart.so.13: cannot open shared object file: No such file or directory error occured when running uv run vllm serve ...

Below are some result of collect_env.py:

CUDA_HOME=/usr/local/cuda-12.9
CUDA_HOME=/usr/local/cuda-12.9
LD_LIBRARY_PATH=/usr/local/cuda-12.9/compat:/usr/local/cuda-12.9/targets/x86_64-linux/lib/:/usr/local/cuda-12.9/lib64:
VLLM_CUDA_COMPATIBILITY_PATH=/usr/local/cuda-12.9/compat

9

If you still get ImportError: libcudart.so.13: cannot open shared object file after installing cuda-compat-12-9 and setting LD_LIBRARY_PATH and VLLM_CUDA_COMPATIBILITY_PATH, it means vLLM or PyTorch is trying to load CUDA 13 libraries, not 12.9. This usually happens if you installed vLLM or torch with cu130/cu131 wheels, not cu129 (vLLM troubleshooting, NVIDIA CUDA compatibility).

Solution:
Uninstall vllm and torch, then reinstall both with cu129 wheels only. For example:

uv pip uninstall vllm torch torchvision torchaudio
uv pip install torch==<version> --index-url https://download.pytorch.org/whl/cu129
uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.19.0/cu129 --extra-index-url https://download.pytorch.org/whl/cu129

Make sure no cu130/cu131 wheels are present in your environment. Would you like more detail on how to check your installed packages?

Sources:

Hint: Mention @RunLLM in the post for followups.

10

Actually I need to install vllm==0.20 to serve DeepSeek-v4, installing whl file from https://wheels.vllm.ai/0.20.0/cu129/vllm works!