I’m currently deploying the Qwen2.5-VL-32B-Instruct-AWQ model using vLLM on a RTX A6000 GPU with CUDA 12.7. The typical workload includes: Average input tokens: ~2000 Average output tokens: ~100 Batch size: 1 VRAM usage: ~44GB Response time: around 5–9 seconds per request The vLLM launch command …

Performance Degradation and Compatibility Issues with AWQ Quantization in vLLM (Qwen2.5-VL-32B)

Topic		Replies	Views
[Field Report] AWQ on RTX 5060 Ti (SM_120 / Blackwell) — awq_marlin + TRITON_ATTN working Model Support	1	58	March 18, 2026
Proper settings for running qwen2.5 72b on 48gb using awq General	1	859	June 21, 2025
A bit of frustration with Quantization Quantization	5	592	October 14, 2025
No throughput improvement for Quantized Qwen-2.5-7B-Instruct? General	4	162	November 13, 2025
GLM 5 / Kimi k2.5 on 4 x RTX 6000 Pro Quantization	1	57	March 22, 2026