# Help with vLLM crashes **URL:** \ **Category:** General\ **Created:** [December 15, 2025, 3:07pm UTC](https://discuss.vllm.ai/t/help-with-vllm-crashes/2120 "2025-12-15T15:07:55Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![crossijinn](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@crossijinn](https://discuss.vllm.ai/u/crossijinn)\ **Post date:** [December 15, 2025, 3:07pm UTC](https://discuss.vllm.ai/t/help-with-vllm-crashes/2120/1 "2025-12-15T15:07:55Z")
Hi, I have the problem that my vLLM crashes from time to time. this is my setup (with 8xH200): vllm: image: vllm/vllm-openai:v0.12.0 ulimits: memlock: soft: -1 hard: -1 stack: soft: 67108864 hard: 67108864 nofile: soft: 1048576 hard: 1048576 shm\_size: 512gb container\_name: vllm restart: unless-stopped ```auto environment: - VLLM_USE_FLASHINFER_MOE_FP16=1 - OMP_NUM_THREADS=2 volumes: - /opt/docker/openwebui/vllm/models:/models:ro - /opt/docker/openwebui/vllm/cache:/root/.cache/huggingface command: - "--model=/models/Qwen3-VL-235B-A22B-Instruct" - "--served-model-name=Delphi" - "--host=0.0.0.0" - "--port=8000" - "--api-server-count=4" - "--tensor-parallel-size=8" - "--async-scheduling" - "--enable-expert-parallel" - "--dtype=bfloat16" - "--kv-cache-dtype=fp8" - "--gpu-memory-utilization=0.7" - "--max-model-len=131072" - "--max-num-batched-tokens=65536" - "--max-num-seqs=32" - "--enable-chunked-prefill" - "--enable-prefix-caching" - "--swap-space=128" - "--mm-encoder-tp-mode=data" - "--mm_processor_cache_type=shm" - "--mm_processor_cache_gb=128" deploy: resources: reservations: devices: - driver: nvidia count: 8 capabilities: ["gpu"] ``` and this is a example of the error: (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [dump\_input.py:72] Dumping input data for V1 LLM engine (v0.11.2) with config: model=‘/models/Qwen3-VL-235B-A22B-Instruct’, speculative\_config=None, tokenizer=‘/models/Qwen3-VL-235B-A22B-Instruct’, skip\_tokenizer\_init=False, tokenizer\_mode=auto, revision=None, tokenizer\_revision=None, trust\_remote\_code=False, dtype=torch.bfloat16, max\_seq\_len=131072, download\_dir=None, load\_format=auto, tensor\_parallel\_size=8, pipeline\_parallel\_size=1, data\_parallel\_size=1, disable\_custom\_all\_reduce=False, quantization=None, enforce\_eager=False, kv\_cache\_dtype=fp8, device\_config=cuda, structured\_outputs\_config=StructuredOutputsConfig(backend=‘auto’, disable\_fallback=False, disable\_any\_whitespace=False, disable\_additional\_properties=False, reasoning\_parser=‘’, reasoning\_parser\_plugin=‘’, enable\_in\_reasoning=False), observability\_config=ObservabilityConfig(show\_hidden\_metrics\_for\_version=None, otlp\_traces\_endpoint=None, collect\_detailed\_traces=None), seed=0, served\_model\_name=Delphi, enable\_prefix\_caching=True, enable\_chunked\_prefill=True, pooler\_config=None, compilation\_config={‘level’: None, ‘mode’: \, ‘debug\_dump\_path’: None, ‘cache\_dir’: ‘’, ‘compile\_cache\_save\_format’: ‘binary’, ‘backend’: ‘inductor’, ‘custom\_ops’: [‘none’], ‘splitting\_ops’: [‘vllm::unified\_attention’, ‘vllm::unified\_attention\_with\_output’, ‘vllm::unified\_mla\_attention’, ‘vllm::unified\_mla\_attention\_with\_output’, ‘vllm::mamba\_mixer2’, ‘vllm::mamba\_mixer’, ‘vllm::short\_conv’, ‘vllm::linear\_attention’, ‘vllm::plamo2\_mamba\_mixer’, ‘vllm::gdn\_attention\_core’, ‘vllm::kda\_attention’, ‘vllm::sparse\_attn\_indexer’], ‘compile\_mm\_encoder’: False, ‘use\_inductor’: None, ‘compile\_sizes’: , ‘inductor\_compile\_config’: {‘enable\_auto\_functionalized\_v2’: False, ‘combo\_kernels’: True, ‘benchmark\_combo\_kernel’: True}, ‘inductor\_passes’: {}, ‘cudagraph\_mode’: \, ‘cudagraph\_num\_of\_warmups’: 1, ‘cudagraph\_capture\_sizes’: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], ‘cudagraph\_copy\_inputs’: False, ‘cudagraph\_specialize\_lora’: True, ‘use\_inductor\_graph\_partition’: False, ‘pass\_config’: {}, ‘max\_cudagraph\_capture\_size’: 64, ‘local\_cache\_dir’: None}, (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [dump\_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled\_new\_reqs=[NewRequestData(req\_id=chatcmpl-50cfb5507c4c401590f28cb0cfcc65da,prompt\_token\_ids\_len=83338,mm\_features=,sampling\_params=SamplingParams(n=1, presence\_penalty=0.0, frequency\_penalty=0.0, repetition\_penalty=1.0, temperature=0.7, top\_p=0.8, top\_k=20, min\_p=0.0, seed=None, stop=, stop\_token\_ids=[151643], bad\_words=, include\_stop\_str\_in\_output=False, ignore\_eos=False, max\_tokens=47734, min\_tokens=0, logprobs=None, prompt\_logprobs=None, skip\_special\_tokens=True, spaces\_between\_special\_tokens=True, truncate\_prompt\_tokens=None, structured\_outputs=None, extra\_args=None),block\_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 41991, 41992, 41993, 41994, 41995, 41996, 41997, 41998, 41999, 42000, 42001, 42002, 42003, 42004, 42005, 42006, 42007, 42008, 42009, 42010, 42011, 42012, 42013, 42014, 42015, 42016, 42017, 42018, 42019, 42020, 42021, 42022, 42023, 42024, 42025, 42026, 42027, 42028, 42029, 42030, 42031, 42032, 42033, 46086],),num\_computed\_tokens=320,lora\_request=None,prompt\_embeds\_shape=None)], scheduled\_cached\_reqs=CachedRequestData(req\_ids=, resumed\_req\_ids=, new\_token\_ids=, all\_token\_ids={}, new\_block\_ids=, num\_computed\_tokens=, num\_output\_tokens=), num\_scheduled\_tokens={chatcmpl-50cfb5507c4c401590f28cb0cfcc65da: 65536}, total\_num\_scheduled\_tokens=65536, scheduled\_spec\_decode\_tokens={}, scheduled\_encoder\_inputs={}, num\_common\_prefix\_blocks=[4116], finished\_req\_ids=, free\_encoder\_mm\_hashes=, pending\_structured\_output\_tokens=false, kv\_connector\_metadata=null, ec\_connector\_metadata=null) (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [dump\_input.py:81] Dumping scheduler stats: SchedulerStats(num\_running\_reqs=1, num\_waiting\_reqs=0, step\_counter=0, current\_wave=0, kv\_cache\_usage=0.0745250085841822, prefix\_cache\_stats=PrefixCacheStats(reset=False, requests=1, queries=83338, hits=320, preempted\_requests=0, preempted\_queries=0, preempted\_hits=0), connector\_prefix\_cache\_stats=None, spec\_decoding\_stats=None, kv\_connector\_stats=None, waiting\_lora\_adapters={}, running\_lora\_adapters={}) (EngineCore\_DP0 pid=18) [2025-12-12 01:44:15] ERROR \_base.py:342: exception calling callback for \ (EngineCore\_DP0 pid=18) Traceback (most recent call last): (EngineCore\_DP0 pid=18) File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 335, in get\_response (EngineCore\_DP0 pid=18) status, result = mq.dequeue( (EngineCore\_DP0 pid=18) ^^^^^^^^^^^ (EngineCore\_DP0 pid=18) File “/usr/local/lib/python3.12/dist-packages/vllm/distributed/device\_communicators/shm\_broadcast.py”, line 571, in dequeue (EngineCore\_DP0 pid=18) with self.acquire\_read(timeout, cancel, indefinite) as buf: (EngineCore\_DP0 pid=18) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) File “/usr/lib/python3.12/contextlib.py”, line 137, in **enter** (EngineCore\_DP0 pid=18) return next(self.gen) (EngineCore\_DP0 pid=18) ^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) File “/usr/local/lib/python3.12/dist-packages/vllm/distributed/device\_communicators/shm\_broadcast.py”, line 495, in acquire\_read (EngineCore\_DP0 pid=18) raise TimeoutError (EngineCore\_DP0 pid=18) TimeoutError (EngineCore\_DP0 pid=18) (EngineCore\_DP0 pid=18) The above exception was the direct cause of the following exception: (EngineCore\_DP0 pid=18) (EngineCore\_DP0 pid=18) Traceback (most recent call last): (EngineCore\_DP0 pid=18) File “/usr/lib/python3.12/concurrent/futures/\_base.py”, line 340, in \_invoke\_callbacks (EngineCore\_DP0 pid=18) callback(self) (EngineCore\_DP0 pid=18) File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 322, in callback (EngineCore\_DP0 pid=18) result = f.result() (EngineCore\_DP0 pid=18) ^^^^^^^^^^ (EngineCore\_DP0 pid=18) File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 79, in result (EngineCore\_DP0 pid=18) return super().result() (EngineCore\_DP0 pid=18) ^^^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) File “/usr/lib/python3.12/concurrent/futures/\_base.py”, line 449, in result (EngineCore\_DP0 pid=18) return self.\_\_get\_result() (EngineCore\_DP0 pid=18) ^^^^^^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) File “/usr/lib/python3.12/concurrent/futures/\_base.py”, line 401, in \_\_get\_result (EngineCore\_DP0 pid=18) raise self.\_exception (EngineCore\_DP0 pid=18) File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 83, in wait\_for\_response (EngineCore\_DP0 pid=18) response = self.aggregate(get\_response()) (EngineCore\_DP0 pid=18) ^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 339, in get\_response (EngineCore\_DP0 pid=18) raise TimeoutError(f"RPC call to {method} timed out.“) from e (EngineCore\_DP0 pid=18) TimeoutError: RPC call to execute\_model timed out. (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [dump\_input.py:72] Dumping input data for V1 LLM engine (v0.11.2) with config: model=‘/models/Qwen3-VL-235B-A22B-Instruct’, speculative\_config=None, tokenizer=‘/models/Qwen3-VL-235B-A22B-Instruct’, skip\_tokenizer\_init=False, tokenizer\_mode=auto, revision=None, tokenizer\_revision=None, trust\_remote\_code=False, dtype=torch.bfloat16, max\_seq\_len=131072, download\_dir=None, load\_format=auto, tensor\_parallel\_size=8, pipeline\_parallel\_size=1, data\_parallel\_size=1, disable\_custom\_all\_reduce=False, quantization=None, enforce\_eager=False, kv\_cache\_dtype=fp8, device\_config=cuda, structured\_outputs\_config=StructuredOutputsConfig(backend=‘auto’, disable\_fallback=False, disable\_any\_whitespace=False, disable\_additional\_properties=False, reasoning\_parser=‘’, reasoning\_parser\_plugin=‘’, enable\_in\_reasoning=False), observability\_config=ObservabilityConfig(show\_hidden\_metrics\_for\_version=None, otlp\_traces\_endpoint=None, collect\_detailed\_traces=None), seed=0, served\_model\_name=Delphi, enable\_prefix\_caching=True, enable\_chunked\_prefill=True, pooler\_config=None, compilation\_config={‘level’: None, ‘mode’: \, ‘debug\_dump\_path’: None, ‘cache\_dir’: ‘’, ‘compile\_cache\_save\_format’: ‘binary’, ‘backend’: ‘inductor’, ‘custom\_ops’: [‘none’], ‘splitting\_ops’: [‘vllm::unified\_attention’, ‘vllm::unified\_attention\_with\_output’, ‘vllm::unified\_mla\_attention’, ‘vllm::unified\_mla\_attention\_with\_output’, ‘vllm::mamba\_mixer2’, ‘vllm::mamba\_mixer’, ‘vllm::short\_conv’, ‘vllm::linear\_attention’, ‘vllm::plamo2\_mamba\_mixer’, ‘vllm::gdn\_attention\_core’, ‘vllm::kda\_attention’, ‘vllm::sparse\_attn\_indexer’], ‘compile\_mm\_encoder’: False, ‘use\_inductor’: None, ‘compile\_sizes’: , ‘inductor\_compile\_config’: {‘enable\_auto\_functionalized\_v2’: False, ‘combo\_kernels’: True, ‘benchmark\_combo\_kernel’: True}, ‘inductor\_passes’: {}, ‘cudagraph\_mode’: \, ‘cudagraph\_num\_of\_warmups’: 1, ‘cudagraph\_capture\_sizes’: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], ‘cudagraph\_copy\_inputs’: False, ‘cudagraph\_specialize\_lora’: True, ‘use\_inductor\_graph\_partition’: False, ‘pass\_config’: {}, ‘max\_cudagraph\_capture\_size’: 64, ‘local\_cache\_dir’: None}, (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [dump\_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled\_new\_reqs=[NewRequestData(req\_id=chatcmpl-50cfb5507c4c401590f28cb0cfcc65da,prompt\_token\_ids\_len=83338,mm\_features=,sampling\_params=SamplingParams(n=1, presence\_penalty=0.0, frequency\_penalty=0.0, repetition\_penalty=1.0, temperature=0.7, top\_p=0.8, top\_k=20, min\_p=0.0, seed=None, stop=, stop\_token\_ids=[151643], bad\_words=, include\_stop\_str\_in\_output=False, ignore\_eos=False, max\_tokens=47734, min\_tokens=0, logprobs=None, prompt\_logprobs=None, skip\_special\_tokens=True, spaces\_between\_special\_tokens=True, truncate\_prompt\_tokens=None, structured\_outputs=None, extra\_args=None),block\_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 41991, 41992, 41993, 41994, 41995, 41996, 46068, 46069, 46070, 46071, 46072, 46073, 46074, 46075, 46076, 46077, 46078, 46079, 46080, 46081, 46082, 46083, 46084, 46085, 46086],),num\_computed\_tokens=320,lora\_request=None,prompt\_embeds\_shape=None)], scheduled\_cached\_reqs=CachedRequestData(req\_ids=, resumed\_req\_ids=, new\_token\_ids=, all\_token\_ids={}, new\_block\_ids=, num\_computed\_tokens=, num\_output\_tokens=), num\_scheduled\_tokens={chatcmpl-50cfb5507c4c401590f28cb0cfcc65da: 65536}, total\_num\_scheduled\_tokens=65536, scheduled\_spec\_decode\_tokens={}, scheduled\_encoder\_inputs={}, num\_common\_prefix\_blocks=[4116], finished\_req\_ids=, free\_encoder\_mm\_hashes=, pending\_structured\_output\_tokens=false, kv\_connector\_metadata=null, ec\_connector\_metadata=null) (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [dump\_input.py:81] Dumping scheduler stats: SchedulerStats(num\_running\_reqs=1, num\_waiting\_reqs=0, step\_counter=0, current\_wave=0, kv\_cache\_usage=0.0745250085841822, prefix\_cache\_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted\_requests=0, preempted\_queries=0, preempted\_hits=0), connector\_prefix\_cache\_stats=None, spec\_decoding\_stats=None, kv\_connector\_stats=None, waiting\_lora\_adapters={}, running\_lora\_adapters={}) (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] EngineCore encountered a fatal error. (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] Traceback (most recent call last): (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 335, in get\_response (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] status, result = mq.dequeue( (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/distributed/device\_communicators/shm\_broadcast.py”, line 571, in dequeue (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] with self.acquire\_read(timeout, cancel, indefinite) as buf: (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/lib/python3.12/contextlib.py”, line 137, in **enter** (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] return next(self.gen) (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/distributed/device\_communicators/shm\_broadcast.py”, line 495, in acquire\_read (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] raise TimeoutError (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] TimeoutError (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] The above exception was the direct cause of the following exception: (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] Traceback (most recent call last): (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 835, in run\_engine\_core (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] engine\_core.run\_busy\_loop() (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 862, in run\_busy\_loop (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] self.\_process\_engine\_step() (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 891, in \_process\_engine\_step (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] outputs, model\_executed = self.step\_fn() (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py”, line 437, in step\_with\_batch\_queue (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] model\_output = future.result() (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 79, in result (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] return super().result() (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/lib/python3.12/concurrent/futures/\_base.py”, line 449, in result (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] return self.\_\_get\_result() (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/lib/python3.12/concurrent/futures/\_base.py”, line 401, in \_\_get\_result (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] raise self.\_exception (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 83, in wait\_for\_response (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] response = self.aggregate(get\_response()) (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] ^^^^^^^^^^^^^^ (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc\_executor.py”, line 339, in get\_response (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] raise TimeoutError(f"RPC call to {method} timed out.”) from e (EngineCore\_DP0 pid=18) ERROR 12-12 01:44:15 [core.py:844] TimeoutError: RPC call to sample\_tokens timed out. (Worker\_TP1\_EP1 pid=53) INFO 12-12 01:44:15 [multiproc\_executor.py:702] Parent process exited, terminating worker (Worker\_TP2\_EP2 pid=54) INFO 12-12 01:44:15 [multiproc\_executor.py:702] Parent process exited, terminating worker (Worker\_TP5\_EP5 pid=57) INFO 12-12 01:44:15 [multiproc\_executor.py:702] Parent process exited, terminating worker (Worker\_TP6\_EP6 pid=58) INFO 12-12 01:44:15 [multiproc\_executor.py:702] Parent process exited, terminating worker (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] AsyncLLM output\_handler failed. (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] Traceback (most recent call last): (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 477, in output\_handler (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] outputs = await engine\_core.get\_output\_async() (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core\_client.py”, line 883, in get\_output\_async (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] raise self.\_format\_exception(outputs) from None (ApiServer\_1 pid=20) ERROR 12-12 01:44:15 [async\_llm.py:525] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] AsyncLLM output\_handler failed. (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] Traceback (most recent call last): (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 477, in output\_handler (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] outputs = await engine\_core.get\_output\_async() (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core\_client.py”, line 883, in get\_output\_async (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] raise self.\_format\_exception(outputs) from None (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [async\_llm.py:525] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] AsyncLLM output\_handler failed. (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] Traceback (most recent call last): (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 477, in output\_handler (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] outputs = await engine\_core.get\_output\_async() (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core\_client.py”, line 883, in get\_output\_async (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] raise self.\_format\_exception(outputs) from None (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [async\_llm.py:525] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] AsyncLLM output\_handler failed. (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] Traceback (most recent call last): (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 477, in output\_handler (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] outputs = await engine\_core.get\_output\_async() (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core\_client.py”, line 883, in get\_output\_async (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] raise self.\_format\_exception(outputs) from None (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [async\_llm.py:525] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] Error in chat completion stream generator. (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] Traceback (most recent call last): (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving\_chat.py”, line 611, in chat\_completion\_stream\_generator (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] async for res in result\_generator: (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 423, in generate (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] out = q.get\_nowait() or await q.get() (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] ^^^^^^^^^^^^^ (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output\_processor.py”, line 70, in get (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] raise output (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 477, in output\_handler (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] outputs = await engine\_core.get\_output\_async() (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core\_client.py”, line 883, in get\_output\_async (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] raise self.\_format\_exception(outputs) from None (ApiServer\_2 pid=21) ERROR 12-12 01:44:15 [serving\_chat.py:1278] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] Error in chat completion stream generator. (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] Traceback (most recent call last): (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving\_chat.py”, line 611, in chat\_completion\_stream\_generator (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] async for res in result\_generator: (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 423, in generate (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] out = q.get\_nowait() or await q.get() (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] ^^^^^^^^^^^^^ (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output\_processor.py”, line 70, in get (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] raise output (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 477, in output\_handler (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] outputs = await engine\_core.get\_output\_async() (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core\_client.py”, line 883, in get\_output\_async (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] raise self.\_format\_exception(outputs) from None (ApiServer\_0 pid=19) ERROR 12-12 01:44:15 [serving\_chat.py:1278] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] Error in chat completion stream generator. (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] Traceback (most recent call last): (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving\_chat.py”, line 611, in chat\_completion\_stream\_generator (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] async for res in result\_generator: (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 423, in generate (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] out = q.get\_nowait() or await q.get() (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] ^^^^^^^^^^^^^ (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output\_processor.py”, line 70, in get (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] raise output (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async\_llm.py”, line 477, in output\_handler (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] outputs = await engine\_core.get\_output\_async() (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] File “/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core\_client.py”, line 883, in get\_output\_async (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] raise self.\_format\_exception(outputs) from None (ApiServer\_3 pid=22) ERROR 12-12 01:44:15 [serving\_chat.py:1278] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [December 16, 2025, 10:28am UTC](https://discuss.vllm.ai/t/help-with-vllm-crashes/2120/2 "2025-12-16T10:28:27Z")
Your vLLM instance is crashing due to a TimeoutError in the multiprocessing executor, which is often caused by requests that are too large (e.g., very long prompts or outputs) or by resource exhaustion (e.g., not enough GPU memory for the requested batch/context size). The logs show a request with 83,338 prompt tokens and 47,734 output tokens, which is extremely large and likely exceeds what the model and hardware can handle reliably, even with 8xH200 and a reduced `--gpu-memory-utilization` of 0.7. This can cause the engine to hang or crash, resulting in EngineDeadError and timeouts as seen in your logs. To mitigate this, reduce `--max-model-len`, `--max-num-batched-tokens`, and/or `--max-num-seqs` to limit the maximum request size, and consider enforcing stricter input validation on the client side to reject overly large requests. For more troubleshooting strategies and memory conservation tips, see the official vLLM troubleshooting and memory documentation ([troubleshooting](https://docs.vllm.ai/en/latest/usage/troubleshooting/), [conserving memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory/)). Would you like a step-by-step breakdown or specific parameter recommendations? Sources: - [Troubleshooting - vLLM](https://docs.vllm.ai/en/latest/usage/troubleshooting/) - [conserving memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory/) Hint: Mention @RunLLM in the post for followups.