How can I use vllm with cuda 12.9

General
1

image
image2187×750 42.8 KB

I’m working on an Ubuntu 22.04 x86_64 environment with Python 3.12 and NVIDIA CUDA 12.9, trying to install and run vLLM 0.24.0.

I hit this import error:

ImportError: libcudart.so.13: cannot open shared object file: No such file or directory

The program is looking for CUDA 130 binaries, but according to the official docs, vLLM should default to CUDA 129.

I’ve already tried setting the environment variable:

export VLLM_PRECOMPILED_WHEEL_VARIANT=cu129

I also tried the installation flag --torch-backend=cu129.

Yet the installed wheel still has dependencies targeting CUDA 130.

Is there no way to install a vLLM 0.24.0 build compiled for CUDA 129 at all?

2

There is currently no official vLLM 0.24.0 release wheel compiled for CUDA 12.9 (cu129)—the default PyPI wheel is built for CUDA 13.0 (cu130), and the documented install commands (including --torch-backend=cu129 or setting VLLM_PRECOMPILED_WHEEL_VARIANT=cu129) do not fetch a cu129-compatible vLLM wheel for 0.24.0. This is confirmed by both user reports and maintainers: the vLLM 0.24.0 release wheel links against CUDA 13, and the cu129-specific index does not provide a 0.24.0 wheel, so the install always resolves to the CUDA 13 build, causing the libcudart.so.13 error on CUDA 12.9 systems (see discussion, see also).

Workarounds:

  • You can either (1) upgrade your system to CUDA 13.0, or (2) build vLLM 0.24.0 from source against CUDA 12.9 as described in the official build instructions and here.
    Would you like detailed step-by-step instructions for building from source?

Sources:

Hint: Mention @Herald in the post for followups.

5

I change to use 0.22.0 to solve my problem. After comparing our solutions, the only difference between yours and mine is that you installed the CUDA Toolkit. Will the setup work successfully once this toolkit is installed?

8

I think the “extra-index-url” can solve your issue that saying libcudart.so.13 required. This will ensure you downloaded the wheel built with cuda129 instead of cuda130 by default. The torch-backend=cu129 only make sure the pytorch installed is cu129 verison.

I am not sure with the cuda-toolkit for your case. Because I am using docker containers. . When I run with cuda runtime (12.9), it pops some compiling error message when initiating models (this did not happen when I was using v0.19.0). And the official image has toolkit installed so I did it as well.

uv pip install vllm==0.24.0 \
–torch-backend=cu129 \
–extra-index-url https://wheels.vllm.ai/0.24.0/cu129/ \
–index-strategy unsafe-best-match \
–system