I tried to enable LMFE_STRICT_JSON_FIELD_ORDER = true in the env vars in Runpod but it still allows the LLM to output json in whatever order it wants (such as outputting the actual response first and the reasoning behind it second).
Outlines is also supposed to enforce order, but doesn’t in practice.
h ttps://github.com/runpod-workers/worker-vllm/issues/233
Setting LMFE_STRICT_JSON_FIELD_ORDER=true in the environment variables for LM Format Enforcer (LMFE) should enforce the JSON field order according to the ‘required’ list in your JSON schema, but some users report that it does not always work as expected, and the LLM may still output fields in a different order or mix reasoning and response content. This is a known limitation, and similar issues have been observed with Outlines, which is also supposed to enforce field order but may not do so reliably in practice. See discussion in vLLM issue #3283 and LMFE configuration docs.
Would you like a step-by-step breakdown of why this happens and possible workarounds?
Sources:
Hint: Mention @RunLLM in the post for followups.
I thought the issue was that the worker-vllm doesn’t support vLLM’s structured outputs correctly, but I then confirmed that it does, when using the OpenAI API, and that vLLM structured outputs are just buggy:
If I give it a system prompt to output YAML and then constrain it to json schema, the constraint does override the system prompt and it generates valid json, so it is constraining the tokens generated. But it only produces valid JSON most of time. Sometimes it fails, like putting Byte Order Mark first, or adding a closing bracket without closing the double quotation mark, etc. so it’s not working 100%.