Currently, vLLM disables CUDA graphs and torch.compile when using data parallelism (--data-parallel-size) because CUDA graphs are not yet supported in this mode. This is due to technical limitations: CUDA graphs require static execution and memory management, but data parallel engines are separate processes, making it difficult to safely share or reuse compiled graphs and their memory across processes. As a result, vLLM enforces eager mode for stability and correctness in DP setups[^1][^2].
Sharing compilation artifacts (like torch.compile graphs) across engines is a desired feature, but it’s non-trivial due to process isolation and device context differences. The vLLM team is aware of this limitation and may address it in future releases, but for now, eager mode is required for data parallel serving. Would you like more technical detail on the challenges or current roadmap?
Sources: