# How to benchmark concurrency **URL:** \ **Category:** General\ **Created:** [September 15, 2025, 6:52am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611 "2025-09-15T06:52:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 6:52am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/1 "2025-09-15T06:52:57Z")
Hi, How should I benchmark vLLM docker setup with 2 GPUs in ubuntu 24.04. I want to know how many concurrent users in certain prompt it can handle? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 6:53am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/2 "2025-09-15T06:53:19Z")
To benchmark your vLLM Docker setup with 2 GPUs and measure how many concurrent users it can handle for a given prompt, use the built-in benchmark scripts included in the vLLM Docker image. Start the vLLM server with your model and desired parallelism (e.g., `--tensor-parallel-size 2`), then run the `benchmark_serving.py` script from inside the container to simulate concurrent requests and measure throughput, latency, and concurrency. You can control the number of concurrent requests using the `--max-concurrency` flag in the benchmark script, which was added for this purpose. For example: ```bash python benchmarks/benchmark_serving.py --backend openai --base-url http://localhost:8000 --num-prompts 100 --max-concurrency 32 ``` This will report metrics such as request throughput, token throughput, and latency, helping you determine the maximum concurrency your setup can handle for your prompt size. For more details, see the [official benchmarking documentation](https://docs.vllm.ai/en/latest/api/vllm/benchmarks/serve.html) and [max concurrency support](https://github.com/vllm-project/vllm/pull/9390). Benchmarks are included in the Docker image, so you do not need to clone the repo separately unless using a custom image. Would you like a step-by-step example for your specific model and prompt? Sources: - [vllm.benchmarks.serve - vLLM](https://docs.vllm.ai/en/latest/api/vllm/benchmarks/serve.html) - [benchmark: Add option to set max concurrency](https://github.com/vllm-project/vllm/pull/9390) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:04am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/3 "2025-09-15T07:04:05Z")
ubuntu@ubuntuAIhome01:~$ docker exec -it vllm\_download\_and\_serve10 bash root@ubuntuAIhome01:/vllm-workspace# python benchmarks/benchmark\_serving.py --backend openai --base-url [http://localhost:8000](http://localhost:8000) --num-prompts 100 --max-concurrency 32 bash: python: command not found root@ubuntuAIhome01:/vllm-workspace# python3 benchmarks/benchmark\_serving.py --backend openai --base-url [http://localhost:8000](http://localhost:8000) --num-prompts 100 --max-concurrency 32 Traceback (most recent call last): File “/vllm-workspace/benchmarks/benchmark\_serving.py”, line 59, in from benchmark\_dataset import ( File “/vllm-workspace/benchmarks/benchmark\_dataset.py”, line 28, in import pandas as pd ModuleNotFoundError: No module named ‘pandas’ ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 7:04am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/4 "2025-09-15T07:04:27Z")
The error means the pandas Python package is not installed in your Docker container. To fix this, run: ```bash pip install pandas ``` Then rerun your benchmark command. This is a common requirement, as the benchmark scripts depend on pandas for data handling, but it is not always pre-installed in the vLLM Docker image according to [GitHub Issue #19078](https://github.com/vllm-project/vllm/issues/19078). Would you like more troubleshooting steps or help with other missing dependencies? Sources: - [GitHub Issue #19078](https://github.com/vllm-project/vllm/issues/19078) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:05am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/5 "2025-09-15T07:05:10Z")
File “/vllm-workspace/benchmarks/benchmark\_serving.py”, line 59, in from benchmark\_dataset import ( File “/vllm-workspace/benchmarks/benchmark\_dataset.py”, line 29, in from datasets import load\_dataset ModuleNotFoundError: No module named ‘datasets’ ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 7:05am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/6 "2025-09-15T07:05:31Z")
You are missing the datasets package, which is required by the benchmark scripts. Install it with: ```bash pip install datasets ``` This is expected, as benchmark dependencies like datasets are not always included by default in the vLLM Docker image, per [GitHub Issue #19078](https://github.com/vllm-project/vllm/issues/19078). Would you like a list of all recommended benchmark dependencies? Sources: - [GitHub Issue #19078](https://github.com/vllm-project/vllm/issues/19078) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:06am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/7 "2025-09-15T07:06:10Z")
INFO 09-15 00:05:56 [**init**.py:241] Automatically detected platform cuda. usage: benchmark\_serving.py [-h] [–backend {tgi,vllm,lmdeploy,deepspeed-mii,openai,openai-chat,openai-audio,tensorrt-llm,scalellm,sglang,llama.cpp}] [–base-url BASE\_URL] [–host HOST] [–port PORT] [–endpoint ENDPOINT] [–dataset-name {sharegpt,burstgpt,sonnet,random,hf,custom}] [–dataset-path DATASET\_PATH] [–no-stream] [–max-concurrency MAX\_CONCURRENCY] --model MODEL [–tokenizer TOKENIZER] [–use-beam-search] [–num-prompts NUM\_PROMPTS] [–logprobs LOGPROBS] [–request-rate REQUEST\_RATE] [–burstiness BURSTINESS] [–seed SEED] [–trust-remote-code] [–disable-tqdm] [–profile] [–save-result] [–save-detailed] [–append-result] [–metadata [KEY=VALUE …]] [–result-dir RESULT\_DIR] [–result-filename RESULT\_FILENAME] [–ignore-eos] [–percentile-metrics PERCENTILE\_METRICS] [–metric-percentiles METRIC\_PERCENTILES] [–goodput GOODPUT [GOODPUT …]] [–custom-output-len CUSTOM\_OUTPUT\_LEN] [–custom-skip-chat-template] [–sonnet-input-len SONNET\_INPUT\_LEN] [–sonnet-output-len SONNET\_OUTPUT\_LEN] [–sonnet-prefix-len SONNET\_PREFIX\_LEN] [–sharegpt-output-len SHAREGPT\_OUTPUT\_LEN] [–random-input-len RANDOM\_INPUT\_LEN] [–random-output-len RANDOM\_OUTPUT\_LEN] [–random-range-ratio RANDOM\_RANGE\_RATIO] [–random-prefix-len RANDOM\_PREFIX\_LEN] [–hf-subset HF\_SUBSET] [–hf-split HF\_SPLIT] [–hf-output-len HF\_OUTPUT\_LEN] [–top-p TOP\_P] [–top-k TOP\_K] [–min-p MIN\_P] [–temperature TEMPERATURE] [–tokenizer-mode {auto,slow,mistral,custom}] [–served-model-name SERVED\_MODEL\_NAME] [–lora-modules LORA\_MODULES [LORA\_MODULES …]] [–ramp-up-strategy {linear,exponential}] [–ramp-up-start-rps RAMP\_UP\_START\_RPS] [–ramp-up-end-rps RAMP\_UP\_END\_RPS] benchmark\_serving.py: error: the following arguments are required: --model ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 7:06am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/8 "2025-09-15T07:06:44Z")
The error means you must specify the --model argument when running benchmark\_serving.py. For example: ```bash python3 benchmarks/benchmark_serving.py --backend openai --base-url http://localhost:8000 --num-prompts 100 --max-concurrency 32 --model ``` Replace `` with the name or path of your model. This is a required argument as documented in the [benchmarking API](https://docs.vllm.ai/en/latest/api/vllm/benchmarks/serve.html). Would you like help choosing the correct model name or path? Sources: - [https://github.com/vllm-project/vllm/pull/21095](https://github.com/vllm-project/vllm/pull/21095) - [https://github.com/vllm-project/vllm/pull/20528](https://github.com/vllm-project/vllm/pull/20528) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:09am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/9 "2025-09-15T07:09:08Z")
The model is already loaded into gpu vram and its running in this same docker. Anyway this just does not work: DeprecationWarning: benchmark\_serving.py is deprecated and will be removed in a future version. Please use ‘vllm bench serve’ instead. main(args) Namespace(backend=‘openai’, base\_url=‘[http://localhost:8000](http://localhost:8000)’, host=‘127.0.0.1’, port=8000, endpoint=‘/v1/completions’, dataset\_name=‘sharegpt’, dataset\_path=None, no\_stream=False, max\_concurrency=32, model=‘google/gemma-3-12b-it’, tokenizer=None, use\_beam\_search=False, num\_prompts=100, logprobs=None, request\_rate=inf, burstiness=1.0, seed=0, trust\_remote\_code=False, disable\_tqdm=False, profile=False, save\_result=False, save\_detailed=False, append\_result=False, metadata=None, result\_dir=None, result\_filename=None, ignore\_eos=False, percentile\_metrics=‘ttft,tpot,itl’, metric\_percentiles=‘99’, goodput=None, custom\_output\_len=256, custom\_skip\_chat\_template=False, sonnet\_input\_len=550, sonnet\_output\_len=150, sonnet\_prefix\_len=200, sharegpt\_output\_len=None, random\_input\_len=1024, random\_output\_len=128, random\_range\_ratio=0.0, random\_prefix\_len=0, hf\_subset=None, hf\_split=None, hf\_output\_len=None, top\_p=None, top\_k=None, min\_p=None, temperature=None, tokenizer\_mode=‘auto’, served\_model\_name=None, lora\_modules=None, ramp\_up\_strategy=None, ramp\_up\_start\_rps=None, ramp\_up\_end\_rps=None) Traceback (most recent call last): File “/vllm-workspace/benchmarks/benchmark\_serving.py”, line 1305, in main(args) File “/usr/local/lib/python3.12/dist-packages/typing\_extensions.py”, line 2956, in wrapper return arg(\*args, \*\*kwargs) ^^^^^^^^^^^^^^^^^^^^ File “/vllm-workspace/benchmarks/benchmark\_serving.py”, line 780, in main input\_requests = dataset\_mappingargs.dataset\_name ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/vllm-workspace/benchmarks/benchmark\_serving.py”, line 759, in “sharegpt”: lambda: ShareGPTDataset( ^^^^^^^^^^^^^^^^ File “/vllm-workspace/benchmarks/benchmark\_dataset.py”, line 384, in **init** self.load\_data() File “/vllm-workspace/benchmarks/benchmark\_dataset.py”, line 388, in load\_data raise ValueError(“dataset\_path must be provided for loading data.”) ValueError: dataset\_path must be provided for loading data. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 7:09am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/10 "2025-09-15T07:09:45Z")
The error is because you set `--dataset-name sharegpt` but did not provide `--dataset-path`. The ShareGPT dataset requires a path to a JSON file (e.g., `ShareGPT_V3_unfiltered_cleaned_split.json`). You must download the dataset and specify its path: ```bash python3 benchmarks/benchmark_serving.py --backend openai --base-url http://localhost:8000 --num-prompts 100 --max-concurrency 32 --model google/gemma-3-12b-it --dataset-name sharegpt --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json ``` Alternatively, to run a synthetic benchmark without a dataset file, use `--dataset-name random` (no `--dataset-path` needed). The script is deprecated; the recommended command is now `vllm bench serve` as described in the [official docs](https://docs.vllm.ai/en/latest/api/vllm/benchmarks/serve.html). Would you like a full example using `vllm bench serve`? Sources: - [benchmarks/README.md](https://github.com/vllm-project/vllm/blob/main/benchmarks/README.md) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:10am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/11 "2025-09-15T07:10:32Z")
How to run benchmark when the docker container and vllm is already running and the model is taking all vram? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 7:10am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/12 "2025-09-15T07:10:56Z")
You can run the benchmark directly inside the same Docker container where vLLM is serving the model, even if the model is using all GPU VRAM. The benchmark script (`vllm bench serve` or `python3 benchmarks/benchmark_serving.py`) only sends HTTP requests to the running vLLM server and does not load the model itself, so it does not require additional GPU memory. Just ensure you have the required Python dependencies installed (e.g., `pandas`, `datasets`). Example (inside the container): ```bash vllm bench serve --backend openai --base-url http://localhost:8000 --model --num-prompts 100 --max-concurrency 32 --dataset-name random ``` This approach is documented in the [official benchmarking guide](https://docs.vllm.ai/en/latest/api/vllm/benchmarks/serve.html) and confirmed in [GitHub Issue #13598](https://github.com/vllm-project/vllm/issues/13598). Would you like a step-by-step example for your setup? Sources: - [GitHub Issue #13598](https://github.com/vllm-project/vllm/issues/13598) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:31am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/13 "2025-09-15T07:31:34Z")
trying to server local model for the benchmark while inside the vllm docker: vllm bench serve –backend openai –base-url [http://localhost:8000](http://localhost:8000) –endpoint /v1/chat/completions –model vllm/gemma-3 –num-prompts 100 –max-concurrency 32 –dataset-name random OSError: vllm/gemma-3 is not a local folder and is not a valid model identifier listed on ‘[https://huggingface.co/models’](https://huggingface.co/models%E2%80%99) If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `hf auth login` or by passing `token=` I run the docker with -v /home/ubuntu/vllm\_models:/root/.cache/huggingface so the models are outside of the docker in the host ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 7:32am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/14 "2025-09-15T07:32:23Z")
The error means `vllm/gemma-3` is not a valid Hugging Face repo or a direct path to a local model directory. When serving a local model, you must provide the absolute path to the model directory containing `config.json` (e.g., `/root/.cache/huggingface/gemma-3` or `/root/.cache/huggingface/hub/models----/snapshots/`). Just mounting `/home/ubuntu/vllm_models` is not enough; you must use the correct path inside the container. Example: ```bash vllm bench serve --backend openai --base-url http://localhost:8000 --endpoint /v1/chat/completions --model /root/.cache/huggingface/gemma-3 --num-prompts 100 --max-concurrency 32 --dataset-name random ``` If you downloaded the model with `huggingface-cli`, the path is usually `/root/.cache/huggingface/hub/models----/snapshots/`. See [this issue](https://github.com/vllm-project/vllm/issues/9459) and [this comment](https://github.com/vllm-project/vllm/issues/1131) for details. Would you like help finding the exact model path inside your container? Sources: - [GitHub Issue #9459](https://github.com/vllm-project/vllm/issues/9459) - [GitHub Issue #1131](https://github.com/vllm-project/vllm/issues/1131) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:33am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/15 "2025-09-15T07:33:26Z")
huggingface\_hub.errors.HFValidationError: Repo id must be in the form ‘repo\_name’ or ‘namespace/repo\_name’: ‘/root/.cache/huggingface/gemma-3’. Use `repo_type` argument if needed. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 7:34am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/16 "2025-09-15T07:34:11Z")
This error means `/root/.cache/huggingface/gemma-3` is not the correct directory—vLLM expects the path to the model snapshot directory that contains `config.json`, not just a parent folder. For Hugging Face-downloaded models, the correct path is usually: ```auto /root/.cache/huggingface/hub/models----/snapshots/ ``` Make sure you point `--model` to this snapshot directory, not a higher-level folder. See [GitHub Issue #9459](https://github.com/vllm-project/vllm/issues/9459) for details. Would you like help locating the exact snapshot path for your model? Sources: - [GitHub Issue #9459](https://github.com/vllm-project/vllm/issues/9459) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 7:34am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/17 "2025-09-15T07:34:50Z")
> [@RunLLM](#): > > Would you like help locating the exact snapshot path for your model? yes, the model is already runnin. why it is so hard to find the model path? ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 8:10am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/18 "2025-09-15T08:10:07Z")
with this command: vllm bench serve --backend openai --base-url [http://localhost:8000](http://localhost:8000) --endpoint /v1/chat/completions --model google/gemma-3-12b-it --served-model-name vllm/gemma-3 --num-prompts 100 --max-concurrency 32 --dataset-name random I get this: File “/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/serve.py”, line 396, in benchmark raise ValueError( ValueError: Initial test run failed - Please make sure benchmark arguments are correctly specified. Error: Bad Request ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 8:10am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/19 "2025-09-15T08:10:40Z")
This error usually means the `--model` argument in your benchmark command does not match the `--served-model-name` used when you started the vLLM server. The model name in your benchmark request must exactly match the name the server is serving under, not the Hugging Face repo or local path. For example, if you started the server with: ```bash vllm serve /root/.cache/huggingface/hub/models--google--gemma-3-12b-it/snapshots/ --served-model-name vllm/gemma-3 ``` then your benchmark command should use: ```bash vllm bench serve --backend openai --base-url http://localhost:8000 --endpoint /v1/chat/completions --model vllm/gemma-3 --num-prompts 100 --max-concurrency 32 --dataset-name random ``` If the names do not match, you will get a “Bad Request” or “Not Found” error, as confirmed in [GitHub Issue #19743](https://github.com/vllm-project/vllm/issues/19743) and [GitHub Issue #19047](https://github.com/vllm-project/vllm/issues/19047). Would you like step-by-step instructions to verify your served model name? Sources: - [GitHub Issue #19743](https://github.com/vllm-project/vllm/issues/19743) - [GitHub Issue #19047](https://github.com/vllm-project/vllm/issues/19047) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 8:12am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/20 "2025-09-15T08:12:40Z")
That exactly does not work. This is how I started the vllm: docker run -it --gpus all --ipc=host --network host -p 8000:8000 -v /home/ubuntu/vllm\_models:/root/.cache/huggingface -e HUGGING\_FACE\_HUB\_TOKEN=“” --name vllm\_download\_and\_serve10 vllm/vllm-openai:latest --model google/gemma-3-12b-it --served-model-name vllm/gemma-3 --max-model-len 2048 --max-num-seqs 128 --gpu-memory-utilization 0.96 and that works from outside. [Next page](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611.md?page=2)