What is the best way to support vLLM on a custom hardware?

General
1

Hello,

I have seen that we can support custom hardware using a platform plugin. If a custom hardware already has support for torch backend, can we avoid implementing a platform plugin?

2

No, you cannot avoid implementing a platform plugin even if your custom hardware supports the torch backend. vLLM requires a platform plugin to register and manage hardware-specific components (e.g., executor, worker, model runner, attention backend, communicator, and custom ops), regardless of torch compatibility. The plugin system enables vLLM to recognize, initialize, and dispatch to your hardware in a modular and maintainable way, as detailed in the official plugin documentation and RFCs. Without a platform plugin, vLLM will not detect or utilize your hardware backend automatically, even if torch operations work on it natively. See vLLM’s Plugin System and Hardware pluggable RFC.

Would you like more detail on how to implement a platform plugin for your hardware?

Sources:

Hint: Mention @RunLLM in the post for followups.

1 Like