I checked all my code and made sure nothing I did today could have impacted this. No changes in version, nothing. But try after try, it just kept outputting this kind of garbage.
Out of desperation, I shutdown vllm and turned it back on. Lo and behold, it works like a charm again.
So now I’m really confused, are we supposed to refresh vllm once in a while? Could long running sessions corrupt memory in a way that harm the way it works?
It is possible for long-running vLLM sessions to encounter issues such as memory corruption, resource leaks, or state inconsistencies that can affect output quality or tool parsing. Restarting vLLM can clear these transient issues and restore normal behavior, as observed in your case. This is not an officially required maintenance step, but similar problems have been reported and resolved by restarting the server, especially after heavy or prolonged workloads, or when encountering unexpected output or crashes (source, source, source).
If you notice degraded behavior, a restart is a reasonable troubleshooting step. For persistent or frequent issues, check logs for errors, enable debug logging, and ensure you are using the latest vLLM version, as many bugs are fixed in newer releases (source). Would you like more detail on best practices for long-running vLLM deployments or troubleshooting steps?