Mistral Small 4: ValueError: grammar is not of valid supported types. (StructuredOutputOptions.LARK)

Model Support
1

We are discovering an issue with vLLM and Mistral Small 4 119B, which we host and provide for a larger user base.

We tried to use regular vLLM Containers, however the generated and returned text was sometimes odd with strange characters and responses. So, we switched to the Mistral 4 container (mistralllm/vllm-ms4 - Docker Image) as was more stable.

In the last few days, although we did not change anything, we discovered an issue which lead to vLLM crashing with the following error message:

(EngineCore pid=134) ERROR 10-02 09:57:07 [backend_xgrammar.py:108] Validation should have already occurred. Please file an issue.
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101] EngineCore encountered a fatal error.^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101] Traceback (most recent call last):^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/engine/core.py", line 1092, in run_engine_core^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     engine_core.run_busy_loop()^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/engine/core.py", line 1133, in run_busy_loop^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     self._process_engine_step()^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/engine/core.py", line 1172, in _process_engine_step^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     outputs, model_executed = self.step_fn()^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]                               ^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/engine/core.py", line 447, in step_with_batch_queue^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     scheduler_output = self.scheduler.schedule()^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/core/sched/scheduler.py", line 570, in schedule^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     ) and not self._try_promote_blocked_waiting_request(request):^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/core/sched/scheduler.py", line 2069, in _try_promote_blocked_waiting_request^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     if not (structured_output_req and structured_output_req.grammar):^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/structured_output/request.py", line 54, in grammar^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     completed = self._check_grammar_completion()^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/structured_output/request.py", line 42, in _check_grammar_completion^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     self._grammar = self._grammar.result(timeout=0.0001)^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     return self.__get_result()^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]            ^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     raise self._exception^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     result = self.fn(*self.args, **self.kwargs)^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/structured_output/__init__.py", line 165, in _create_grammar^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     return self.backend.compile_grammar(request_type, grammar_spec)^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]   File "/workspace/vllm/vllm/v1/structured_output/backend_xgrammar.py", line 111, in compile_grammar^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101]     raise ValueError(^M
(EngineCore pid=134) ERROR 10-02 09:57:07 [core.py:1101] ValueError: grammar is not of valid supported types. (StructuredOutputOptions.LARK)
(Worker_TP1 pid=150) INFO 10-02 09:57:07 [multiproc_executor.py:759] Parent process exited, terminating worker queues
(Worker_TP0 pid=149) INFO 10-02 09:57:07 [multiproc_executor.py:759] Parent process exited, terminating worker queues
(Worker_TP1 pid=150) INFO 10-02 09:57:07 [multiproc_executor.py:854] WorkerProc shutting down.
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707] AsyncLLM output_handler failed.^M
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707] Traceback (most recent call last):^M
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707]   File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler^M
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707]     outputs = await engine_core.get_output_async()^M
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707]   File "/workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async^M
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707]     raise self._format_exception(outputs) from None^M
(APIServer pid=110) ERROR 10-02 09:57:07 [async_llm.py:707] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(Worker_TP0 pid=149) INFO 10-02 09:57:07 [multiproc_executor.py:854] WorkerProc shutting down.
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388] Error in chat completion stream generator.^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388] Traceback (most recent call last):^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/entrypoints/openai/chat_completion/serving.py", line 625, in chat_completion_stream_generator^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     async for res in result_generator:^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     out = q.get_nowait() or await q.get()^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]                             ^^^^^^^^^^^^^^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     raise output^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/entrypoints/utils.py", line 116, in wrapper^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     return await func(*args, **kwargs)^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/entrypoints/openai/chat_completion/api_router.py", line 55, in create_chat_completion^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     generator = await handler.create_chat_completion(request, raw_request)^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/entrypoints/openai/chat_completion/serving.py", line 379, in create_chat_completion^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     return await self.chat_completion_full_generator(^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/entrypoints/openai/chat_completion/serving.py", line 1411, in chat_completion_full_generator^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     async for res in result_generator:^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     out = q.get_nowait() or await q.get()^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]                             ^^^^^^^^^^^^^^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     raise output^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     outputs = await engine_core.get_output_async()^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]   File "/workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388]     raise self._format_exception(outputs) from None^M
(APIServer pid=110) ERROR 10-02 09:57:07 [serving.py:1388] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

Any idea what can cause such an error? Even if this is related to a user query, it should not lead to a complete crash of the served instanced.

We use the following start parameters

vllm serve --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser mistral --reasoning-parser mistral --disable-log-stats --disable-uvicorn-access-log mistralai/Mistral-Small-4-119B-2603

I’m glad for any advice.

2

Your error is caused by vLLM’s structured output backend (xgrammar) receiving a grammar type (StructuredOutputOptions.LARK) that it does not support, leading to a fatal crash. The error message ValueError: grammar is not of valid supported types. (StructuredOutputOptions.LARK) indicates that an unsupported grammar type was passed, and the code path expects validation to have already filtered this out, but it did not—hence the explicit request to “file an issue” in the log message. This is a bug in vLLM’s error handling: user input or model output should not be able to crash the engine in this way, even if the grammar is invalid or unsupported.

This type of crash has been reported before and is related to structured outputs and grammar validation not being robust enough, especially when using tool calling or reasoning parsers with Mistral models. The root cause is not a user query, but a mismatch between the expected grammar backend and the actual grammar type provided, which should be handled gracefully by vLLM instead of causing a fatal error. See similar issues and discussions in the vLLM repo for context and possible workarounds, such as disabling structured outputs or ensuring only supported grammar types are used in your requests (source, source).

Would you like more detail on how to mitigate or debug this further?

Sources:

Hint: Mention @Herald in the post for followups.