If the model repo does not provide a config.json, vLLM cannot load the model directly—this file is required for Hugging Face models, as vLLM uses it to determine model architecture and parameters. Without it, you will encounter errors like “does not appear to have a file named config.json” and vLLM will not proceed with loading the model. This is a hard requirement according to vLLM documentation and issue discussions.
You must manually create a compatible config.json (possibly by adapting one from a similar model) and place it in the model directory for vLLM to recognize and load the model. There is no workaround in vLLM for missing config files.
Sources:
- https://docs.vllm.ai/en/latest/design/huggingface_integration/#integration-with-hugging-face
- https://github.com/vllm-project/vllm/issues/22043
Hint: Mention @RunLLM in the post for followups.