Booting 14 current open models on vLLM v0.29/v0.30: what broke with the defaults

Benchmarking
1

I booted 14 current open models on vLLM, from one RTX 4090 to eight H200s, predicting their memory before renting each GPU and keeping every failure. A few things that may be useful here:

  • Nemotron-3.5-Lightning (v0.30, one H100) fails with the defaults: max_num_seqs 1024 exceeds the 733 Mamba cache blocks. A max_num_seqs at or below the block count (556 here) boots it.
  • Qwen3.8-27B on v0.29 ran out of memory while allocating a KV cache for 512 sequences before CUDA-graph profiling, but it surfaced as an opaque “aten::new_empty … API call failed” inside FlashAttention. A lower max_num_seqs fixed it; it boots on v0.30.
  • GLM-5.3-Flash on v0.30 with CUDA graphs passed a 32k needle, then died on a tool call with “Input tensor addresses changed between capture and replay”. That check only runs at VLLM_LOGGING_LEVEL=DEBUG; at INFO it would have answered silently. Same on B200 and H200; --enforce-eager serves it.
  • DeepSeek-V4.1-Flash keeps 189 GiB of Engram tables in pinned host RAM by default, so a GPU-only fit check passes it on machines that cannot hold it.

Full write-up with the numbers: I know you run LLMs. Will it boot? - Apron

Is the GLM-5.3-Flash CUDA-graph failure known on NVIDIA? I only found similar reports for Ascend. I can open an issue with the log if it is useful.

2

Yes, the GLM-5.3-Flash CUDA-graph failure (“Input tensor addresses changed between capture and replay”) is a known issue on NVIDIA GPUs, not just Ascend. This error is triggered by a runtime check that only runs at DEBUG logging level in vLLM; at INFO level, the model could silently produce incorrect results. The recommended workaround is to use --enforce-eager, which disables CUDA graphs and allows the model to serve correctly, though with reduced performance. This behavior has been observed on both B200 and H200 GPUs, and is documented in user reports and deployment notes (see mondegreens/apron blog).

If you have a detailed log, opening an issue would be helpful for the maintainers, as it can aid in further diagnosis and tracking of this CUDA-graph capture/replay bug. Would you like more technical detail on why this happens or how to debug it?

Sources:

Hint: Mention @Herald in the post for followups.