Hi all,
We’re running Qwen3.8-27B on vLLM for 700+ users, and our benchmarks showed a performance knee at around 128 concurrent requests with the default settings.
Since vLLM already handles scheduling and queuing dynamically based on available resources, is there any real benefit to explicitly setting max_num_seqs? Wouldn’t leaving it at the default be better unless benchmarks show otherwise?
Explicitly setting max_num_seqs can significantly impact throughput and latency, especially at high concurrency. The default (often 128 or 256) may not be optimal for your hardware/model; increasing it can improve performance if your GPU has enough memory, but setting it too high can cause preemption or OOM. Benchmarking different values is recommended to find the best setting for your workload—leaving it at default is not always optimal, especially for large-scale deployments like 700+ users. See discussion and evidence in this issue and this comment.
Would you like a step-by-step guide on how to tune max_num_seqs for your setup?
Sources:
Hint: Mention @Herald in the post for followups.