I’m running a vLLM server inside a container and using VLLM_CACHE_ROOT to specify the cache directory.
On the container’s first startup, after launching the vLLM server for the first time, the latency of the first request is around 4 seconds.
Later, within the same container, I changed VLLM_CACHE_ROOT to a completely new, empty directory and restarted the server. Surprisingly, the first-request latency was only around 200 ms.
This suggests that changing VLLM_CACHE_ROOT alone does not fully reproduce the original cold-start behavior.
Are there other caches, initialization artifacts, or process-/host-level state that vLLM relies on outside VLLM_CACHE_ROOT? For example, could CUDA initialization, CUDA graph compilation/caching, kernel compilation caches, model-related caches, or other GPU/runtime state remain available after the first server run?
Yes, vLLM relies on additional caches and runtime state outside of VLLM_CACHE_ROOT. Specifically, CUDA kernel compilation artifacts (such as Triton and FlashInfer JIT caches) are stored in locations like ~/.triton/cache and may persist across server restarts within the same container, unless explicitly cleared. Also, GPU driver-level state (e.g., loaded kernels, CUDA context, and CUDA graphs) can remain in memory as long as the container and its processes are alive, which can make subsequent startups and first requests faster even if you change VLLM_CACHE_ROOT to a new directory. This explains why your second “cold” start was much faster: the underlying GPU/runtime state and non-vLLM caches were still warm and available to the process. See torch.compile integration, cache directory configuration, and docker cache persistence.
Would you like more detail on how to fully reproduce a true cold start, including clearing all relevant caches and runtime state?