For example, if my model parameter type is int4 and the LORA parameter matrix type is float16, can I directly configure the LORA matrix to obtain the result?
Yes, you can use a LoRA adapter (FP16) with an INT4 quantized model in vLLM. vLLM materializes FP16 weights from the INT4 packed buffers specifically to enable LoRA attachment, while maintaining INT4 inference efficiency. This hybrid INT4+LoRA setup is supported for both dense and MoE models, and LoRA operates on activations, not directly on the quantized weights, so the types are compatible and results are correct without extra steps or conversions required by the user. See usage and architecture details in Add INT4 + LoRA support with tensor materialization and Add INT4 compressed-tensors + LoRA support (including MoE).
Would you like more detail or example code?
Sources:
- Add INT4 + LoRA support with tensor materialization
- Add INT4 compressed-tensors + LoRA support (including MoE)
Hint: Mention @RunLLM in the post for followups.