Some requests never complete: token generation keeps running but the response never finishes (looks like an infinite generation loop)

General
1

I’m new to vLLM and self-hosting LLMs in general, so please bear with me if I’ve missed anything obvious.

I’m serving QuantTrio/Qwen3.5-35B-A3B-AWQ using vllm/vllm-openai:v0.31.0 on two RTX 4090 GPUs with 24 GB VRAM each, using tensor parallelism (TP=2).

The workload consists of document analysis requests that vary in content but are similar in length and task type. Each request contains approximately 30,000–50,000 input tokens. Responses are typically 500–700 tokens long, occasionally approaching 1,000 tokens, but rarely exceeding that. Occasionally, a request never completes: token generation continues on the server, but the client never receives a completed response. This looks like an infinite generation loop, although I have not confirmed the cause.

The affected requests are comparable to successful ones in both input length and content, with no obvious differences that would explain this behavior.

Deployment configuration:

version: '3.8'

services:
  vllm:
    container_name: vllm
    image: vllm/vllm-openai:v0.31.0
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - PYTORCH_ALLOC_CONF=expandable_segments:True
      - OMP_NUM_THREADS=1
      - VLLM_CACHE_ROOT=/root/.cache/vllm
    volumes:
      - ./.cache/huggingface:/root/.cache/huggingface
      - ./.cache/vllm:/root/.cache/vllm
    ports:
      - "8000:8000"
    command: >
      QuantTrio/Qwen3.5-35B-A3B-AWQ
      --api-key ${VLLM_API_KEY}
      --served-model-name Qwen3.5-35B
      --max-num-seqs 12
      --enable-prefix-caching
      --enable-chunked-prefill
      --max_num_batched_tokens 8132
      --kv-cache-dtype fp8_e4m3
      --tensor-parallel-size 2
      --kv-cache-memory 8846363034
      --max-model-len 131072
      --no-enforce-eager
      --language-model-only
    restart: unless-stopped
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]

Under normal conditions, a single request takes about 15 seconds on average to complete. With 12 concurrent requests, response times range from 60 to 100 seconds. For affected requests, I have waited up to approximately 20 minutes before cancelling them, with token generation still ongoing.
The issue persists even when the affected request is the only running request and no other requests are waiting, suggesting that contention with other requests is not the cause. However, generation throughput remains around 182–188 tokens/s, so this does not appear to be a complete engine stall.

I will attach logs from both normal processing and the problematic interval.

Normal behavior - requests complete successfully and very quickly:

(APIServer pid=1) INFO 10-22 13:15:58 [loggers.py:320] Engine 000: Avg prompt throughput: 11486.2 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 7 reqs, Waiting: 24 reqs, Deferred: 24 reqs, GPU KV cache usage: 11.3%, Prefix cache hit rate: 9.6%
INFO:     10.8.11.2:37104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:16:08 [loggers.py:320] Engine 000: Avg prompt throughput: 16073.9 tokens/s, Avg generation throughput: 152.2 tokens/s, Running: 12 reqs, Waiting: 18 reqs, Deferred: 18 reqs, GPU KV cache usage: 21.5%, Prefix cache hit rate: 9.4%
INFO:     10.8.11.2:37120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:16:18 [loggers.py:320] Engine 000: Avg prompt throughput: 15664.8 tokens/s, Avg generation throughput: 118.8 tokens/s, Running: 11 reqs, Waiting: 18 reqs, Deferred: 18 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 9.3%
INFO:     10.8.11.2:37134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:16:28 [loggers.py:320] Engine 000: Avg prompt throughput: 13782.7 tokens/s, Avg generation throughput: 131.6 tokens/s, Running: 12 reqs, Waiting: 19 reqs, Deferred: 19 reqs, GPU KV cache usage: 22.2%, Prefix cache hit rate: 9.2%
INFO:     10.8.11.2:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:16:38 [loggers.py:320] Engine 000: Avg prompt throughput: 13786.2 tokens/s, Avg generation throughput: 169.8 tokens/s, Running: 12 reqs, Waiting: 19 reqs, Deferred: 19 reqs, GPU KV cache usage: 22.2%, Prefix cache hit rate: 9.1%
INFO:     10.8.11.2:37184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:16:48 [loggers.py:320] Engine 000: Avg prompt throughput: 13784.8 tokens/s, Avg generation throughput: 185.1 tokens/s, Running: 12 reqs, Waiting: 19 reqs, Deferred: 19 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 9.0%
INFO:     10.8.11.2:37104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37370 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:16:58 [loggers.py:320] Engine 000: Avg prompt throughput: 13776.4 tokens/s, Avg generation throughput: 203.4 tokens/s, Running: 12 reqs, Waiting: 19 reqs, Deferred: 19 reqs, GPU KV cache usage: 22.2%, Prefix cache hit rate: 9.0%
INFO:     10.8.11.2:37320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37286 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:34218 - "GET /health HTTP/1.1" 200 OK
INFO:     10.8.11.2:37220 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37206 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:17:08 [loggers.py:320] Engine 000: Avg prompt throughput: 16077.8 tokens/s, Avg generation throughput: 149.2 tokens/s, Running: 12 reqs, Waiting: 18 reqs, Deferred: 18 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 8.9%
INFO:     10.8.11.2:37140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:17:18 [loggers.py:320] Engine 000: Avg prompt throughput: 13915.4 tokens/s, Avg generation throughput: 140.8 tokens/s, Running: 12 reqs, Waiting: 19 reqs, Deferred: 19 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 8.6%
INFO:     10.8.11.2:37190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37134 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37344 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     10.8.11.2:37250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 10-22 13:17:28 [loggers.py:320] Engine 000: Avg prompt throughput: 16070.4 tokens/s, Avg generation throughput: 139.5 tokens/s, Running: 12 reqs, Waiting: 19 reqs, Deferred: 19 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 8.5%

Problematic behavior — token generation continues, but the request never completes:

(APIServer pid=1) INFO 10-22 13:09:08 [loggers.py:320] Engine 000: Avg prompt throughput: 142.8 tokens/s, Avg generation throughput: 60.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:09:18 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:09:28 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:09:38 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 186.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:09:48 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 186.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:09:58 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 185.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:10:08 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:10:18 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:10:28 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:10:38 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:10:48 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:10:58 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 182.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 9.7%
(APIServer pid=1) INFO 10-22 13:11:08 [loggers.py:320] Engine 000: Avg prompt throughput: 137.0 tokens/s, Avg generation throughput: 160.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:11:18 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:11:28 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:11:38 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:11:48 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 186.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:11:58 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:12:08 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:12:18 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:12:28 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:12:38 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:12:48 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:12:58 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 182.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:13:08 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 146.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:13:18 [loggers.py:320] Engine 000: Avg prompt throughput: 137.0 tokens/s, Avg generation throughput: 179.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:13:28 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 187.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:13:38 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 186.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.2%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:13:48 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 186.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:13:58 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 185.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:14:08 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:14:18 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:14:28 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 184.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:14:38 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:14:48 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 183.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:14:58 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 182.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 9.8%
(APIServer pid=1) INFO 10-22 13:15:08 [loggers.py:320] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 181.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 9.8%

Interestingly, if I cancel an affected request after 30 seconds and immediately resubmit the exact same request, it eventually completes successfully. Sometimes it takes 2–3 attempts; other times, 5–7.

What diagnostics would you recommend to determine why these requests keep generating instead of completing? Are there any known issues with this model and configuration?

2

Your symptoms—requests that never complete, with generation throughput remaining high and no engine stall—match known vLLM issues where infinite generation loops occur, especially with quantized or AWQ models and long contexts. This is documented for Qwen3.5 and similar models, where the model can get stuck generating repeated tokens (e.g., exclamation marks or newlines) until max tokens is reached, but never emits a stop sequence or EOS token. This is not always deterministic and can be triggered by certain prompt/model interactions, not just by request size or concurrency. A known workaround is to set --max_num_batched_tokens equal to --max-model-len, which sometimes prevents the issue, but this is not a guaranteed fix for all cases. There are also reports that resubmitting the same request can eventually succeed, as you observed, suggesting the bug is intermittent and context-dependent. See vLLM issue #53305, #16151, and #24038.

For diagnostics, enable DEBUG logging (VLLM_LOGGING_LEVEL=DEBUG) and set VLLM_LOG_STATS_INTERVAL=1 to get more frequent stats. You can also try disabling chunked prefill or prefix caching to see if the issue persists, as these features sometimes interact poorly with certain quantized models. If the problem continues, consider testing with a non-AWQ version of the model or a different quantization backend. There is no universal fix in the data sources, but these steps can help isolate the cause or provide more actionable logs for reporting upstream. Would you like a step-by-step breakdown of these troubleshooting steps?

Sources:

Hint: Mention @Herald in the post for followups.