# vLLM extremely slow / no response with max\_model\_len=8192 and multi-GPU tensor parallel 

**URL:** <https://discuss.vllm.ai/t/vllm-extremely-slow-no-response-with-max-model-len-8192-and-multi-gpu-tensor-parallel/1771>\
**Category:** KV-Cache\
**Created:** [October 26, 2025, 4:01am UTC](https://discuss.vllm.ai/t/vllm-extremely-slow-no-response-with-max-model-len-8192-and-multi-gpu-tensor-parallel/1771 "2025-10-26T04:01:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![infinity-void6](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/infinity-void6/32/821_2.png) [@infinity-void6](https://discuss.vllm.ai/u/infinity-void6)\
**Post date:** [October 26, 2025, 4:01am UTC](https://discuss.vllm.ai/t/vllm-extremely-slow-no-response-with-max-model-len-8192-and-multi-gpu-tensor-parallel/1771/1 "2025-10-26T04:01:54Z")

</div>

**Setup:**

```auto
- Model: llama-3.1-8b

- Hardware: 2x NVIDIA A40

- CUDA: 12.5, Driver: 555.42.06

- vLLM version: 0.10.1.1

```

**- Serving command:**

```auto
CUDA\_VISIBLE\_DEVICES=0,1 vllm serve ./llama-3.1-8b \

--tensor-parallel-size 2 \

--max-model-len 8192 \

--gpu-memory-utilization 0.9 \

--chat-template /opt/vllm\_templates/llama-chat.jinja \

--guided-decoding-backend outlines \

--host [0.0.0.0](http://0.0.0.0) \

--port 9000 \

--max-num-seqs 20

```

**Problem:**

- With max\_model\_len=4096 and top\_k (top\_k is number of chunks/docs retrieved) =2 in my semantic retrieval pipeline → works fine.

- With max\_model\_len=8192, multi-GPU TP=2, top\_k=5 (top\_k is number of chunks/docs retrieved) → server never returns an answer.

- Logs show extremely low throughput:

```auto
Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.2 tokens/s

GPU KV cache usage: 0.4%, Prefix cache hit rate: 66.4%

```

- Context size is ~2800–4000 tokens.

**What I’ve tried:**

- Reduced max\_model\_len → works

- Reduced top\_k(top\_k is number of chunks/docs retrieved)→ works

- Checked GPU memory → not fully used

**Questions:**

1. Is this a known KV cache / memory allocation bottleneck for long contexts in vLLM?

2. Are there ways to batch token processing or offload KV cache to CPU for large max\_model\_len?

3. Recommended vLLM flags for stable long-context inference on multi-GPU setups?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [October 26, 2025, 4:02am UTC](https://discuss.vllm.ai/t/vllm-extremely-slow-no-response-with-max-model-len-8192-and-multi-gpu-tensor-parallel/1771/2 "2025-10-26T04:02:43Z")

</div>

Yes, this is a known issue: with larger max\_model\_len and higher top\_k (more retrieved docs/chunks), vLLM can become extremely slow or hang, especially in multi-GPU tensor parallel (TP) setups, even if GPU memory is not fully used. This is due to increased activation memory and KV cache requirements, as well as possible inefficiencies in how vLLM schedules and batches long-context requests in TP mode. The logs you see (very low throughput, low KV cache usage) are typical symptoms of this bottleneck. Reducing max\_model\_len or top\_k alleviates the problem because it lowers the memory and scheduling pressure on the engine. See discussion in [vLLM GitHub issue #10102](https://github.com/vllm-project/vllm/issues/10102) and [#13074](https://github.com/vllm-project/vllm/issues/13074).

To improve stability and throughput for long-context, multi-GPU inference: (1) Lower --max-num-seqs to reduce concurrent requests and memory pressure; (2) Tune --max-num-batched-tokens (try 2048–8192) for better batching; (3) Use FP8 KV cache quantization if supported (`--kv-cache-dtype fp8`) to double cache capacity ([Quantized KV Cache docs](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html)); (4) Offloading KV cache to CPU is not fully supported for partial offload—vLLM still requires enough GPU memory for at least one full sequence, and CPU offload is not a complete workaround ([see this discussion](https://github.com/vllm-project/vllm/issues/23087)). For best results, keep max\_model\_len as low as your use case allows, and monitor batch size and token count. Would you like a step-by-step tuning checklist or example command?

Sources:

- [vLLM GitHub issue #10102](https://github.com/vllm-project/vllm/issues/10102)
- [vLLM GitHub issue #13074](https://github.com/vllm-project/vllm/issues/13074)
- [Quantized KV Cache docs](https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html)
- [vLLM GitHub issue #23087](https://github.com/vllm-project/vllm/issues/23087)

Hint: Mention @RunLLM in the post for followups.