Handling request timeouts and KV cache bloat during bursty async workloads in vLLM

We are currently integrating a self-hosted vLLM deployment on an A100 80GB GPU to handle automated text summarization for our automated dachecker pipeline. The workflow processes batches of unstructured web text and domain metadata concurrently, forwarding prompts through the OpenAI-compatible vLLM endpoint using an asynchronous worker.

While steady-state throughput is solid, we run into major latency spikes and eventual request timeouts whenever the queue dumps several dozen requests with wildly different context lengths at once. We suspect our KV cache is getting heavily fragmented due to the fluctuating sequence lengths, which causes subsequent jobs to stall in the waiting queue instead of prefilling efficiently.

Has anyone tackled similar scheduling bottlenecks when feeding variable-length scraping tasks into vLLM? We are wondering whether tweaking chunked prefill or adjusting the swap space allocation would be the best path forward to prevent our ingestion service from dropping active connections during traffic bursts.

Yes, this is a common issue in vLLM when handling mixed-length requests: large/variable context lengths can cause head-of-line blocking and KV cache fragmentation, leading to latency spikes and stalled requests. Enabling chunked prefill (which is on by default in V1) helps by breaking up large prefills and interleaving them with decode requests, improving both throughput and latency. Tuning max_num_batched_tokens (e.g., increasing it to 8192 or higher) can further optimize batching and reduce queueing delays, especially on large GPUs like the A100 80GB. Adjusting swap space (--swap-space) can help if you are running out of GPU memory, but the main lever for your scenario is chunked prefill and batch size tuning, not swap space alone. See Optimization and Tuning and Chunked Prefill for details.

If you still see requests stalling in the waiting queue despite these changes, consider increasing gpu_memory_utilization (e.g., to 0.9) and monitoring KV cache usage. Also, ensure your async worker is saturating the queue with enough parallelism, as uneven producer rates can also cause throughput drops. Would you like a step-by-step guide on tuning these parameters for your workload?

Sources:

Hint: Mention @Herald in the post for followups.