# Num request running stays on 1 **URL:** \ **Category:** Benchmarking\ **Created:** [August 29, 2025, 6:30pm UTC](https://discuss.vllm.ai/t/num-request-running-stays-on-1/1483 "2025-08-29T18:30:22Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![biranofer](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/biranofer/32/665_2.png) [@biranofer](https://discuss.vllm.ai/u/biranofer)\ **Post date:** [August 29, 2025, 6:30pm UTC](https://discuss.vllm.ai/t/num-request-running-stays-on-1/1483/1 "2025-08-29T18:30:22Z")
On a prefill test (with max\_tokens=1) generating multiple requests of lengths 6000 tokens, while max\_num\_batched\_tokens is 60000, max\_num\_seqs is 16. The vllm:num\_requests\_running is always 1, even when vllm:num\_requests\_waiting gets to 30. It runs on A100. The log for a request is: ————————————————————————————– params: SamplingParams(n=1, presence\_penalty=0.0, frequency\_penalty=0.0, repetition\_penalty=1.0, temperature=1.0, top\_p=1.0, top\_k=0, min\_p=0.0, seed=None, stop=, stop\_token\_ids=, bad\_words=, include\_stop\_str\_in\_output=False, ignore\_eos=False, max\_tokens=1, min\_tokens=0, logprobs=None, prompt\_logprobs=None, skip\_special\_tokens=True, spaces\_between\_special\_tokens=True, truncate\_prompt\_tokens=None, guided\_decoding=None, extra\_args=None), ……… ……. prompt\_embeds shape: None, lora\_request: None. ————————————————————————————— This is the server startup log, Any idea ? —————————————————————————————- INFO 08-29 09:22:20 [**init**.py:235] Automatically detected platform cuda. INFO 08-29 09:22:24 [api\_server.py:1755] vLLM API server version 0.10.1.dev1+gbcc0a3cbe INFO 08-29 09:22:24 [cli\_args.py:261] non-default args: {‘model\_tag’: ‘mistralai/Mistral-7B-Instruct-v0.2’, ‘host’: ‘0.0.0.0’, ‘model’: ‘mistralai/Mistral-7B-Instruct-v0.2’, ‘trust\_remote\_code’: True, ‘max\_model\_len’: 16384, ‘block\_size’: 64, ‘enable\_prefix\_caching’: False, ‘max\_num\_batched\_tokens’: 60000, ‘max\_num\_seqs’: 16, ‘enable\_chunked\_prefill’: False} INFO 08-29 09:22:35 [config.py:1604] Using max model len 16384 INFO 08-29 09:22:35 [config.py:2434] Chunked prefill is enabled with max\_num\_batched\_tokens=60000. /usr/local/lib/python3.12/dist-packages/vllm/transformers\_utils/tokenizer\_group.py:24: FutureWarning: It is strongly recommended to run mistral models with `--tokenizer-mode "mistral"` to ensure correct encoding and decoding. self.tokenizer = get\_tokenizer(self.tokenizer\_id, \*\*tokenizer\_config) INFO 08-29 09:22:44 [**init**.py:235] Automatically detected platform cuda. INFO 08-29 09:22:48 [core.py:572] Waiting for init message from front-end. INFO 08-29 09:22:48 [core.py:71] Initializing a V1 LLM engine (v0.10.1.dev1+gbcc0a3cbe) with config: model=‘mistralai/Mistral-7B-Instruct-v0.2’, speculative\_config=None, tokenizer=‘mistralai/Mistral-7B-Instruct-v0.2’, skip\_tokenizer\_init=False, tokenizer\_mode=auto, revision=None, override\_neuron\_config={}, tokenizer\_revision=None, trust\_remote\_code=True, dtype=torch.bfloat16, max\_seq\_len=16384, download\_dir=None, load\_format=LoadFormat.AUTO, tensor\_parallel\_size=1, pipeline\_parallel\_size=1, disable\_custom\_all\_reduce=False, quantization=None, enforce\_eager=False, kv\_cache\_dtype=auto, device\_config=cuda, decoding\_config=DecodingConfig(backend=‘auto’, disable\_fallback=False, disable\_any\_whitespace=False, disable\_additional\_properties=False, reasoning\_backend=‘’), observability\_config=ObservabilityConfig(show\_hidden\_metrics\_for\_version=None, otlp\_traces\_endpoint=None, collect\_detailed\_traces=None), seed=0, served\_model\_name=mistralai/Mistral-7B-Instruct-v0.2, num\_scheduler\_steps=1, multi\_step\_stream\_outputs=True, enable\_prefix\_caching=False, chunked\_prefill\_enabled=True, use\_async\_output\_proc=True, pooler\_config=None, compilation\_config={“level”:3,“debug\_dump\_path”:“”,“cache\_dir”:“”,“backend”:“”,“custom\_ops”:,“splitting\_ops”:[“vllm.unified\_attention”,“vllm.unified\_attention\_with\_output”,“vllm.mamba\_mixer2”],“use\_inductor”:true,“compile\_sizes”:,“inductor\_compile\_config”:{“enable\_auto\_functionalized\_v2”:false},“inductor\_passes”:{},“use\_cudagraph”:true,“cudagraph\_num\_of\_warmups”:1,“cudagraph\_capture\_sizes”:[32,24,16,8,4,2,1],“cudagraph\_copy\_inputs”:false,“full\_cuda\_graph”:false,“max\_capture\_size”:32,“local\_cache\_dir”:null} INFO 08-29 09:22:50 [parallel\_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 INFO 08-29 09:22:50 [topk\_topp\_sampler.py:49] Using FlashInfer for top-p & top-k sampling. INFO 08-29 09:22:50 [gpu\_model\_runner.py:1843] Starting to load model mistralai/Mistral-7B-Instruct-v0.2… INFO 08-29 09:22:50 [gpu\_model\_runner.py:1875] Loading model from scratch… INFO 08-29 09:22:50 [cuda.py:290] Using Flash Attention backend on V1 engine. INFO 08-29 09:22:50 [weight\_utils.py:296] Using model weights format [‘\*.safetensors’] INFO 08-29 09:23:13 [weight\_utils.py:312] Time spent downloading weights for mistralai/Mistral-7B-Instruct-v0.2: 23.020574 seconds > Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00\ > Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:01\<00:03, 1.89s/it] > > Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:03\<00:01, 1.84s/it] > > Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:05\<00:00, 1.87s/it] > > Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:05\<00:00, 1.87s/it] > > INFO 08-29 09:23:19 [default\_loader.py:262] Loading weights took 5.67 seconds > INFO 08-29 09:23:20 [gpu\_model\_runner.py:1892] Model loading took 13.4967 GiB and 29.303647 seconds > INFO 08-29 09:23:27 [backends.py:530] Using cache directory: /tmp/.cache/vllm/torch\_compile\_cache/043061b246/rank\_0\_0/backbone for vLLM’s torch.compile > INFO 08-29 09:23:27 [backends.py:541] Dynamo bytecode transform time: 6.85 s > INFO 08-29 09:23:29 [backends.py:194] Cache the graph for dynamic shape for later use > INFO 08-29 09:23:53 [backends.py:215] Compiling a graph for dynamic shape takes 25.93 s > INFO 08-29 09:24:13 [monitor.py:34] torch.compile takes 32.78 s in total > /usr/local/lib/python3.12/dist-packages/torch/utils/cpp\_extension.py:2356: UserWarning: TORCH\_CUDA\_ARCH\_LIST is not set, all archs for visible cards are included for compilation. > If this is not desired, please set os.environ[‘TORCH\_CUDA\_ARCH\_LIST’]. > warnings.warn( > INFO 08-29 09:24:14 [gpu\_worker.py:255] Available KV cache memory: 19.70 GiB > INFO 08-29 09:24:14 [kv\_cache\_utils.py:833] GPU KV cache size: 161,344 tokens > INFO 08-29 09:24:14 [kv\_cache\_utils.py:837] Maximum concurrency for 16,384 tokens per request: 9.85x > > Capturing CUDA graph shapes: 0%| | 0/7 [00:00\ Capturing CUDA graph shapes: 43%|████▎ | 3/7 [00:00\<00:00, 24.64it/s] > Capturing CUDA graph shapes: 86%|████████▌ | 6/7 [00:00\<00:00, 26.17it/s] > Capturing CUDA graph shapes: 100%|██████████| 7/7 [00:00\<00:00, 25.50it/s] > INFO 08-29 09:24:15 [gpu\_model\_runner.py:2485] Graph capturing finished in 1 secs, took 0.11 GiB > INFO 08-29 09:24:15 [core.py:193] init engine (profile, create kv cache, warmup model) took 54.94 seconds > /usr/local/lib/python3.12/dist-packages/vllm/transformers\_utils/tokenizer\_group.py:24: FutureWarning: It is strongly recommended to run mistral models with `--tokenizer-mode "mistral"` to ensure correct encoding and decoding. > self.tokenizer = get\_tokenizer(self.tokenizer\_id, \*\*tokenizer\_config) ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 29, 2025, 6:31pm UTC](https://discuss.vllm.ai/t/num-request-running-stays-on-1/1483/2 "2025-08-29T18:31:19Z")
The reason `vllm:num_requests_running` is always 1, even with many waiting requests, is that your `max_num_seqs` is set to 16, but each request is 6000 tokens and `max_num_batched_tokens` is 60000. This means only 10 requests (60000 / 6000) can be processed in a batch, but vLLM will only batch up to `max_num_seqs` requests. However, if your requests are long and `max_num_batched_tokens` is the limiting factor, vLLM will only process as many requests as can fit within that token budget. If your requests are arriving faster than they can be processed, the rest will queue, and if the scheduler is not able to batch more than one at a time (due to the token or sequence limits), only one will run at a time. This is expected if the batch is always “full” with a single request, or if the scheduler is not able to batch more due to the configuration or request arrival pattern. See [Performance Tuning with Chunked Prefill](https://docs.vllm.ai/en/latest/performance/optimization.html#performance-tuning-with-chunked-prefill) and [Context length and batch size](https://docs.vllm.ai/en/latest/serving/offline_inference.html#context-length-and-batch-size). If you want more concurrent requests running, try reducing the per-request token length, increasing `max_num_batched_tokens`, or adjusting `max_num_seqs` to better match your workload. Also, ensure chunked prefill is enabled (it is by default in vLLM V1), as this can help with batching and throughput. For more details on how these parameters interact, see [Optimization and Tuning](https://docs.vllm.ai/en/latest/configuration/optimization.html). Would you like a step-by-step breakdown of how vLLM batches requests and how to tune these parameters for your use case? Sources: - [Optimization and Tuning](https://docs.vllm.ai/en/latest/configuration/optimization.html) - [Performance Tuning with Chunked Prefill](https://docs.vllm.ai/en/latest/performance/optimization.html#performance-tuning-with-chunked-prefill) - [Context length and batch size](https://docs.vllm.ai/en/latest/serving/offline_inference.html#context-length-and-batch-size) ---
**Author:** ![biranofer](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/biranofer/32/665_2.png) [@biranofer](https://discuss.vllm.ai/u/biranofer)\ **Post date:** [August 29, 2025, 6:38pm UTC](https://discuss.vllm.ai/t/num-request-running-stays-on-1/1483/3 "2025-08-29T18:38:57Z")
with `max_num_batched_tokens = 60000 and each request length is 6000, and max_num_seqs is 16, more than 1 requests should be processed concurrently !` ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 29, 2025, 6:39pm UTC](https://discuss.vllm.ai/t/num-request-running-stays-on-1/1483/4 "2025-08-29T18:39:52Z")
Yes, with `max_num_batched_tokens=60000`, `max_num_seqs=16`, and each request length 6000 tokens, vLLM should be able to process up to 10 requests concurrently (since 60000 / 6000 = 10, which is less than 16). If only 1 request is running at a time, this is not expected and may indicate a scheduling or configuration issue. Please ensure chunked prefill is enabled (it is by default in vLLM V1), and that no other constraints (like memory or model-specific limits) are preventing batching. See [Performance Tuning with Chunked Prefill](https://docs.vllm.ai/en/latest/performance/optimization.html#performance-tuning-with-chunked-prefill) and [SchedulerConfig docs](https://docs.vllm.ai/en/latest/serving/engine_args.html#vllm.engine.arg_utils-_engine_args_parser-schedulerconfig). Would you like a step-by-step troubleshooting guide to diagnose why batching is not occurring as expected? Sources: - [Performance Tuning with Chunked Prefill](https://docs.vllm.ai/en/latest/performance/optimization.html#performance-tuning-with-chunked-prefill) - [SchedulerConfig docs](https://docs.vllm.ai/en/latest/serving/engine_args.html#vllm.engine.arg_utils-_engine_args_parser-schedulerconfig)