Hi,
We’ve been working on a more general auto-tuning framework for vLLM in the context of large-scale LLM inference for the French public sector:
The project originally started as a fork of the OpenShift PSAP auto-tuner, but it has since evolved quite a bit and is now actively maintained.
The goal is to automatically search for vLLM configurations based on the actual workload and performance objectives rather than manually tuning individual parameters.
Some of the capabilities we currently support include:
- Optuna-based search spaces
- custom and multi-objective optimization
- TTFT / ITL / throughput constraints
- GuideLLM benchmarks
- replay of workload characteristics extracted from production traces
- Prometheus metrics
- local and Ray execution
- tuning of speculative decoding configurations such as MTP, EAGLE3, Medusa and n-gram
I also noticed that vLLM now includes benchmarks/auto_tune, which seems to address a related problem with a simpler search over parameters such as max-num-seqs and max-num-batched-tokens.
I’m curious what direction would make the most sense for the broader vLLM ecosystem.
Would there be interest in some of these more general auto-tuning capabilities?
For example, I’m wondering whether it would make sense to:
- upstream some capabilities into vLLM’s auto-tuning benchmark,
- keep this as a separate ecosystem tool,
- or potentially collaborate more closely around the two approaches.
Feedback on the API, workloads and parameters that would be most useful to tune would also be very welcome.