# How to benchmark concurrency **URL:** \ **Category:** General\ **Created:** [September 15, 2025, 6:52am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611 "2025-09-15T06:52:57Z")\ **Posts on this page:** 7\ **Page:** 2
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 8:13am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/21 "2025-09-15T08:13:08Z")
Your vLLM server is running with `--served-model-name vllm/gemma-3`, so your benchmark command must use `--model vllm/gemma-3` (not the Hugging Face name). Also, make sure you use standard dashes (`-`), not typographic dashes (`–`), in your command—otherwise, arguments may not be parsed correctly. Example benchmark command (inside the container): ```bash vllm bench serve --backend openai --base-url http://localhost:8000 --endpoint /v1/chat/completions --model vllm/gemma-3 --num-prompts 100 --max-concurrency 32 --dataset-name random ``` If you still get “Bad Request,” double-check for any invisible Unicode characters or copy-paste issues in your command. This exact approach is confirmed to work in [GitHub Issue #19743](https://github.com/vllm-project/vllm/issues/19743). Would you like help checking for hidden character issues or a way to verify the served model name via the API? Sources: - [GitHub Issue #19743](https://github.com/vllm-project/vllm/issues/19743) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 8:14am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/22 "2025-09-15T08:14:09Z")
You seem not to understand, with that command it wont work: File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py”, line 54, in main args.dispatch\_function(args) File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/serve.py”, line 21, in cmd main(args) File “/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/serve.py”, line 959, in main return asyncio.run(main\_async(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/runners.py”, line 195, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/runners.py”, line 118, in run return self.\_loop.run\_until\_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3.12/asyncio/base\_events.py”, line 691, in run\_until\_complete return future.result() ^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/benchmarks/serve.py”, line 1002, in main\_async tokenizer = get\_tokenizer(tokenizer\_id, ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/vllm/transformers\_utils/tokenizer.py”, line 239, in get\_tokenizer tokenizer = AutoTokenizer.from\_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization\_auto.py”, line 1049, in from\_pretrained tokenizer\_config = get\_tokenizer\_config(pretrained\_model\_name\_or\_path, \*\*kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization\_auto.py”, line 881, in get\_tokenizer\_config resolved\_config\_file = cached\_file( ^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/transformers/utils/hub.py”, line 321, in cached\_file file = cached\_files(path\_or\_repo\_id=path\_or\_repo\_id, filenames=[filename], \*\*kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/dist-packages/transformers/utils/hub.py”, line 511, in cached\_files raise OSError( OSError: vllm/gemma-3 is not a local folder and is not a valid model identifier listed on ‘[https://huggingface.co/models’](https://huggingface.co/models%E2%80%99) If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `hf auth login` or by passing `token=` ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 8:14am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/23 "2025-09-15T08:14:49Z")
Thank you for the detailed traceback. The root cause is: the benchmark client tries to load the tokenizer using the `--model` value (`vllm/gemma-3`), but this is not a valid Hugging Face repo or a local directory. The benchmark script needs a valid tokenizer path, not just the served model name. **Solution:** You must explicitly provide the tokenizer path using `--tokenizer`, pointing to the local directory containing `tokenizer_config.json` (usually the Hugging Face snapshot directory). For example: ```bash vllm bench serve --backend openai --base-url http://localhost:8000 --endpoint /v1/chat/completions \ --model vllm/gemma-3 \ --tokenizer /root/.cache/huggingface/hub/models--google--gemma-3-12b-it/snapshots/ \ --num-prompts 100 --max-concurrency 32 --dataset-name random ``` This requirement is confirmed in [GitHub Issue #19743](https://github.com/vllm-project/vllm/issues/19743) and [GitHub Issue #8938](https://github.com/vllm-project/vllm/issues/8938). Would you like a command to automatically find the correct tokenizer path inside your container? Sources: - [GitHub Issue #19743](https://github.com/vllm-project/vllm/issues/19743) - [GitHub Issue #8938](https://github.com/vllm-project/vllm/issues/8938) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 8:16am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/24 "2025-09-15T08:16:05Z")
> [@RunLLM](#): > > Would you like a command to automatically find the correct tokenizer path inside your container? of course I need that ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 8:16am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/25 "2025-09-15T08:16:28Z")
To automatically find the correct tokenizer path inside your container, run this command in your Docker shell: ```bash find /root/.cache/huggingface/hub/models--google--gemma-3-12b-it -type d -name "snapshots" -exec find {} -type d \; | while read d; do if [-f "$d/tokenizer_config.json"]; then echo "$d"; fi; done ``` This will print the full path to the snapshot directory containing `tokenizer_config.json`, which you should use as the `--tokenizer` argument in your benchmark command. This approach is based on the Hugging Face cache structure described in [GitHub Issue #9459](https://github.com/vllm-project/vllm/issues/9459). Would you like a ready-to-use benchmark command with the discovered path? Sources: - [https://github.com/vllm-project/vllm/issues/19743](https://github.com/vllm-project/vllm/issues/19743) - [https://github.com/vllm-project/vllm/issues/8570](https://github.com/vllm-project/vllm/issues/8570) ---
**Author:** ![youlearnit](https://avatars.discourse-cdn.com/v4/letter/y/71c47a/32.png) [@youlearnit](https://discuss.vllm.ai/u/youlearnit)\ **Post date:** [September 15, 2025, 8:36am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/26 "2025-09-15T08:36:30Z")
it never worked and after half day trying vllm docker benchmark with your, openai and gemini instructions does not work. Maybe if the model is hosted outside docker in local computer is the reason. thank you for trying, I will never try again. Waste of time. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [September 15, 2025, 8:36am UTC](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611/27 "2025-09-15T08:36:57Z")
I’m sorry for your frustration—vLLM benchmarking with Docker and local models can be tricky due to path and environment issues. Mounting the model directory into Docker should work, but the path inside the container must exactly match what vLLM expects (the snapshot directory containing `config.json` and `tokenizer_config.json`). Many users have faced similar issues, especially with local models and tokenizer paths, as documented in [GitHub Issue #9459](https://github.com/vllm-project/vllm/issues/9459) and [#8938](https://github.com/vllm-project/vllm/issues/8938). If you ever reconsider, I recommend: - Verifying the exact snapshot path inside the container. - Explicitly passing `--tokenizer` with the full path. - Using only standard dashes (`-`) in all commands. Thank you for your feedback, and sorry again for the inconvenience. Sources: - [GitHub Issue #8938](https://github.com/vllm-project/vllm/issues/8938) [Previous page](https://discuss.vllm.ai/t/how-to-benchmark-concurrency/1611.md?page=1)