vLLM Optimizer — automated benchmark-driven configuration tuning for vLLM

Benchmarking
1

Hi everyone,

I’ve been working on an open-source project called vLLM Optimizer.

The idea for the project came to me while I was optimizing a model and found myself repeating the same deployment and benchmarking steps over and over again, making only small configuration changes each time.

I also tried auto-tune-vllm, but I found it difficult to fit into my workflow. That made me want to build my own version focused on being simple to configure, easy to understand, and easy to use.

It currently supports:

  • Baseline vs. optimized configuration comparison
  • Multiple trials, warmups, and retries
  • Random and TPE-based search
  • Tuning vLLM arguments such as max_num_seqs, max_num_batched_tokens, prefix caching, attention backends, and more
  • Multi-GPU / tensor-parallel deployments
  • Reproducing individual trials
  • Interactive HTML reports
  • Using an existing vLLM configuration as the starting point

I’ve tested it on real GPU deployments, including H100 and L40 systems, and many of the features were added after running into problems or unexpected behavior while tuning those systems manually.

I’d especially appreciate feedback from people running vLLM in production or doing performance benchmarking:

  • Are there important vLLM parameters or search spaces I’m missing?
  • Are there metrics you would want to see in the HTML report?
  • Are there workloads where this approach would be particularly useful or problematic?
  • Are there better ways I could structure the benchmarking methodology?

The project is still in active development, so any technical feedback, criticism, bug reports, or ideas are very welcome.

GitHub:

2

Your project addresses a real need—automating vLLM configuration tuning and benchmarking. For completeness, ensure you support tuning these key vLLM parameters: max_num_seqs, max_num_batched_tokens, gpu_memory_utilization, tensor_parallel_size, pipeline_parallel_size, and attention backend selection, as these are the main levers for throughput and latency optimization according to the official optimization guide. Also, consider integrating chunked prefill and prefix caching toggles, as these can significantly impact performance in different workloads.

For benchmarking methodology, vLLM’s own auto_tune.sh script iterates over parameter combinations, measures throughput and latency, and finds the best configuration under constraints—mirroring your approach. Including metrics like TTFT (time to first token), TPOT (time per output token), ITL (inter-token latency), and peak GPU memory usage in your HTML reports would align with vLLM’s benchmarking best practices. For workloads, your tool will be most useful for large-batch, high-throughput, or latency-constrained deployments, but be aware that some features (e.g., chunked prefill) may behave differently depending on model size and hardware.

Would you like a more detailed breakdown of recommended metrics, search spaces, or benchmarking scripts?

Sources:

Hint: Mention @Herald in the post for followups.