How to use local model when using vllm serve?

General
1

How to use local model when using vllm serve?

2

To use a local model with vllm serve, specify the absolute path to the directory containing your model files (including config.json) as the model argument. For example:

vllm serve /path/to/your/local/model

This works as long as the directory structure matches the Hugging Face format and contains all required files. No Hugging Face model ID is needed—just the local path. If you encounter errors, ensure the path is correct and includes config.json and other necessary files.

For Docker, use the same approach:

vllm serve --model /usr/local/model/mylocalmodel

as confirmed in the documentation and issues.

Would you like more detail or troubleshooting steps?

Sources:

3

ValueError: With vllm serve, you should provide the model as a positional argument or in a config file instead of via the --model option.

4

You cannot use the --model option with vllm serve. Instead, provide the model path or name as a positional argument (immediately after serve), or specify it in a config YAML file. For example:

vllm serve /path/to/model

or

vllm serve --config config.yaml

where config.yaml contains a model: field. Using --model will always raise this error, as confirmed in the official documentation and tests.

Would you like more detail or examples?

Sources: