Change management strategy for preventing post-OS-auto-update vllm FTS in containers

General
1

I have an RTX50xx GPU running on a Ubuntu box and I am pretty new to the LLM stack, top to bottom. Due to so many new-to-me pieces in play, I’ve been using Docker to get most of the pieces in order such that vllm can serve on my computer.

Two or maybe three times now in the last ~few months, I’ve gone to do “vllm serve” in my container and all-of-a-sudden it doesn’t work anymore. Seems to be that automatic system updates will update the Linux, but the updates apparently deprecate the container’s ability to run on top of the updated OS.

The first time this happened, I plodded through apt automatic update logs and “reverse engineered” my log text to find what all I might downgrade in attempts at making vllm serve once again inside my existing, “it worked yesterday” docker container. Plodding through update logs and issuing undo’s seems like a massively hacky way to go. There’s got to be a better way (or many better ways, i’d bet)

Second time this happened, I found an updated container that worked with updated drivers from nvidia. This one felt kind of a normal, required maintenance update that needed to be done. Nevertheless, waking up to vllm not serving in the container (now for a 2nd time) got me thinking the whole vllm stack + container was somewhat of a house of cards, subject to fall down at any time with no warning (except i’m sure there’s warnings and i just don’t where to look and might not even know which warnings would mean "your containerized vllm is fixing to break when items X Y and Z update on MM/DD which is coming up in 4 days). This feels like reviewing auto-update logs, except in advance, so “previewing prelogs” lol

I know about version lock/freeze with apt, but oof, what all would i lock to prevent containerize vllm from FTS. I have no clue but I bet it would be dozens of locks, and upon locking at least a few of them, the locked version would create headaches and even vulnerability elsewhere.

Considering the above too-long preface, what strategies are there for preventing changes that prevent vllm from working when it worked “yesterday” or 3 days ago if it’s been 3 days since i last tried, so let’s say "when it worked last time and I (ME, the human) didn’t change anything.

Do we have utilities available that can compare & contrast OS vs container vs GPU (or any other ~critical hardware that’s very easy to change)?

Disclosure: During the last vLLM office hours, I am pretty sure I heard, to paraphrase, that everything but torch is now in normal release versions, so if i do a vllm build with the “using existing torch” flag, I could probably dump Docker (for now, till the next new GPU is released ready for saving princesses but not ready for us) and wrangle vllm with the OS instead of container. And that is likely what I’m going to do, but the next latest, greatest must-have GPU can’t be far off so I figured I should ask "how do we do this, keeping containers in line with the OS and its autoupdates, when I kind of thought containers were supposed to isolate us from having to do that, in general, but i guess bc LLM processing is so hardware-bound that containers either pass that work through to the OS, or, if not a pass-thru scenario, do at least require exactly the versions of things (such as NVML today as the first presenting issue for me)

If the most relevant strategy here is “turn off your OS auto-updates,” I’m game for that, but if that’s the case, what’s the strategy for reviewing and applying updates manually without breaking the container’s ability to operate on the OS as it’s updated?

Thank you for all your guidance RunBot

2

It seems likely that your experience—where vLLM containers suddenly stop working after host OS or driver updates—is a common pain point for GPU-accelerated workloads, especially with new hardware like RTX 50xx. Containers do isolate most dependencies, but GPU drivers and CUDA libraries are tightly coupled between the host and the container, so host updates (especially to NVIDIA drivers or kernel) can break compatibility with the containerized stack. This is not unique to vLLM; it affects all CUDA-based workloads. The official vLLM docs and user discussions recommend using official Docker images and keeping both host drivers and container CUDA versions in sync, but do not provide a utility to automatically compare/contrast OS vs container vs GPU compatibility or to warn in advance of breaking changes. Manual review and version pinning are the main strategies mentioned, but these can be tedious and error-prone for complex stacks (docker docs, github issues, installation guide).

My understanding is that the most robust strategies are:

  1. Pin your host NVIDIA driver and CUDA toolkit versions and only update them when you are ready to rebuild or update your containers.
  2. Use versioned Docker images (not :latest) for vLLM and related software, and rebuild containers if you must update the host drivers.
  3. Snapshot or export working containers (e.g., with docker commit or by saving the image) so you can roll back if needed.
  4. Turn off auto-updates for critical system components (like kernel and NVIDIA drivers), and apply updates manually after testing container compatibility.
  5. There is no out-of-the-box utility to check all compatibility layers, but you can script checks for driver, CUDA, and container versions, and compare them to the requirements in the vLLM docs.
    Would you like more detail on any of these strategies or example scripts for version checking?

Sources:

3

Thanks Runbot. When a new GPU isn’t that new anymore and its supporting software starts to stabilize and become available without the need for prerelease versions of (many) dependencies, is it a good idea to migrate out from the container and operate in a plain ole’ virtual environment? And hmm, would a virtual enviornment have the same issues as containers do where they stop working one day due to some system update? I am guessing virtual environments could allow for your choice between cuda12_8 or cuda12_9, but your OS nvidia driver is your OS nvidia driver is your OS nvidia driver and your virtual environment(s) depend on it and have no ability to ~redecorate (maybe “bridge” is the word) a hardware driver to its previous version, correct, or really bridge a virtual’s previous-version driver to the OS current version driver? bc drivers exist outside the virtualized scope of a virt (from UV or Python etc)? tia runbot

4

It could be possible that, once your GPU and its software stack are stable and supported by standard (non-nightly) releases, moving from Docker to a plain Python virtual environment is a reasonable choice for simplicity and performance. However, a Python virtual environment only isolates Python packages—not system libraries or drivers—so it would still rely on the host’s NVIDIA driver and CUDA runtime (installation guide, github issue).

Though I’m not entirely certain, it might be that virtual environments are just as vulnerable as containers to breakage from host system updates, especially if the update changes the NVIDIA driver or kernel. Virtual environments cannot “bridge” or override the system’s GPU driver; they can only select among compatible CUDA/PyTorch wheels, but the actual driver is always the one installed on the OS. So, if the OS driver changes incompatibly, both containers and virtual environments can break, and only the OS admin can roll back or pin the driver version (installation guide).

Would you like more detail on how to manage or pin driver versions, or on the pros/cons of Docker vs. virtual environments for vLLM?

Sources:

5

Ok good, that sounds good. That’s the behavior I would hope for with the GPU driver. Oh, hmm, CUDA runtime is probably very tightly coupled to the GPU driver. CUDA runtime, though not exactly (just) a driver, is driver-adjacent, so it makes sense that wouldn’t be virtualized same as the driver isn’t virtualized. So, that’s great news! Without the container in play, the two birds in the hand won’t have to match the two birds in the bag bc there is no bag. I reckon it’s time once again to see if can get hte vllm running without the help of the container handling “too many for me” pre-release versions of things. Thank you Runbot i think i love you.

6

You’re correct: the CUDA runtime and the NVIDIA driver are tightly coupled, and both must be compatible with each other and with your hardware. Python virtual environments can manage Python packages (like vLLM, PyTorch, etc.), but not system-level components like the CUDA runtime or GPU driver—those always come from the host OS (installation guide). So, without containers, you only need to ensure your OS driver and CUDA runtime are compatible with your chosen Python packages.

Glad to hear you’re ready to try vLLM outside of Docker! If you keep your OS driver and CUDA toolkit stable, and match your PyTorch/vLLM installs to those, you should have a much smoother experience. Let me know if you need step-by-step guidance for a bare-metal install.

Sources:

7

Thanks for the offer of assitance on a bareMTL install. I have tons of notes from my initial attempts during the first couple months after i bought a blackwell architecture GPU on release day having absolutely no clue what i was getting into. It’s going to be fun to discover whether I was actually doing things right and all the pieces just weren’t in place yet (other than pre-release), or if I had just never quite figured out how to do the bareMTL install. Wish me luck! If you do that. But you’re a bot so you might not make wishes. If that’s the case, I get it. All good!

8

Best of luck with your bare-metal install! For Blackwell GPUs, many users struggled early on due to missing or pre-release support in PyTorch, CUDA, and vLLM, so your difficulties were likely not your fault (github issue). Now that official support is stabilizing, following the latest installation guide should give you a much smoother experience.

If you hit any snags, feel free to ask for help—I’m here to assist, even if I can’t make wishes!

Sources: