The code architecture is changed after half passed year. Is there a newly accepted issue to show how to add a new LLM into vLLM? Maybe include model itself (python), self-defined operators, self-defined distributed strategy, and so on.
There is no single newly accepted issue that comprehensively documents how to add a new LLM to vLLM, including custom model code, operators, and distributed strategies. However, the official documentation provides a detailed, up-to-date guide on integrating a new PyTorch model, including implementing the model, registering it, and supporting custom operators or distributed strategies. The process is straightforward if your model is similar to existing ones, but more complex for new operators or architectures. For distributed execution, vLLM’s architecture is designed for extensibility, and you may need to adapt your model to the uniform constructor and configuration system described in the class hierarchy documentation. For further help, the docs recommend opening a GitHub issue or asking on the developer Slack, but there is no single issue that covers all aspects in one place as of now (Adding a New Model — vLLM, Implementing a Basic Model, Registering a Model to vLLM, Class Hierarchy).
Would you like a step-by-step breakdown of how to add a new model, including custom operators and distributed strategy, based on the current documentation?
Sources:
Thanks for your reply. Is it a human answer, or machine answer? I ask the question just now.
RunLLM is a bot, but I think its answers are fine.
1 Like