Hello,
When serving a Mistral model we need to specify —load_format mistral but according to the doc : "mistral" will load weights from consolidated safetensors files used by Mistral models.
I don’t want to use the HF way by using models saved in cache.
For vllm (v0.10.2) is there a way to load non-consolidated safetensors for Mistral models?
Thanks
Yes, you can load non-consolidated (i.e., standard Hugging Face sharded) safetensors for Mistral models in vLLM by not specifying --load-format mistral. The “mistral” load format is only for consolidated safetensors; if you omit --load-format or set it to “auto” or “safetensors”, vLLM will load the standard sharded safetensors as it does for other Hugging Face models. This is confirmed in the documentation and user reports: “If you don’t set --load-format mistral, then vLLM should already load it like a regular HF Hub model” and “removing [–load-format mistral] fixed the issue” for non-consolidated weights (source, source).
Would you like more detail on how to structure your command or on the file naming conventions?
Sources:
Hint: Mention @RunLLM in the post for followups.
@RunLLM I don’t want to use the Hugging Face way of loading models saved in cache
You can load models from a local directory by specifying the absolute path to the model folder (containing config.json, model weights, etc.) in the model argument, instead of using a Hugging Face model ID. This works as long as the directory structure matches what vLLM expects (docs, docs).
If your model is not in Hugging Face format, you must convert it to the expected format (e.g., using save_pretrained from transformers) before loading. Directly passing an in-memory model object is not supported (docs).
Would you like more detail on directory structure or conversion steps?
Sources:
Hint: Mention @RunLLM in the post for followups.