# Performance Degradation and Compatibility Issues with AWQ Quantization in vLLM (Qwen2.5-VL-32B)

**URL:** <https://discuss.vllm.ai/t/performance-degradation-and-compatibility-issues-with-awq-quantization-in-vllm-qwen2-5-vl-32b/1159>\
**Category:** Features\
**Created:** [July 23, 2025, 4:23pm UTC](https://discuss.vllm.ai/t/performance-degradation-and-compatibility-issues-with-awq-quantization-in-vllm-qwen2-5-vl-32b/1159 "2025-07-23T16:23:26Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ghiaog123](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/ghiaog123/32/511_2.png) [@ghiaog123](https://discuss.vllm.ai/u/ghiaog123)\
**Post date:** [July 23, 2025, 4:23pm UTC](https://discuss.vllm.ai/t/performance-degradation-and-compatibility-issues-with-awq-quantization-in-vllm-qwen2-5-vl-32b/1159/1 "2025-07-23T16:23:26Z")

</div>

I’m currently deploying the **Qwen2.5-VL-32B-Instruct-AWQ** model using **vLLM** on a **RTX A6000** GPU with **CUDA 12.7**. The typical workload includes:

- **Average input tokens** : ~2000
- **Average output tokens** : ~100
- **Batch size** : 1
- **VRAM usage** : ~44GB
- **Response time** : around **5–9 seconds** per request

The vLLM launch command I’m using:

```auto
vllm serve Qwen/Qwen2.5-VL-32B-Instruct-AWQ \
  --port 8000 --host 0.0.0.0 \
  --dtype bfloat16 \
  --api-key cbpai \
  --limit-mm-per-prompt image=5,video=5 \
  --max_model_len 70000

```

However, when I switch to the following configuration:

```auto
vllm serve Qwen/Qwen2.5-VL-32B-Instruct-AWQ \
  --port 8000 --host 0.0.0.0 \
  --dtype float16 \
  --api-key cbpai \
  --limit-mm-per-prompt image=5,video=5 \
  --max_model_len 70000 \
  --gpu_memory_utilization 0.95 \
  --quantization awq

```

the **VRAM usage increases to ~47GB** , and the **response time significantly slows down to 30–40 seconds per request**.

My questions are:

1. Is there an issue with the second configuration that’s causing such a performance drop?
2. When I try to run `--quantization=awq` with `--dtype=bfloat16`, vLLM throws an error stating that `bfloat16 is not supported with awq`. Could you clarify why this combination is not allowed?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [July 23, 2025, 4:23pm UTC](https://discuss.vllm.ai/t/performance-degradation-and-compatibility-issues-with-awq-quantization-in-vllm-qwen2-5-vl-32b/1159/2 "2025-07-23T16:23:58Z")

</div>

AWQ quantization in vLLM only supports float16 (not bfloat16) due to kernel limitations; using --dtype=bfloat16 with --quantization=awq will result in an error, as documented in multiple issues and confirmed by error messages in the logs ([source](https://github.com/vllm-project/vllm/issues/1114), [source](https://github.com/vllm-project/vllm/issues/12404), [source](https://github.com/vllm-project/vllm/issues/11122)). The performance drop and increased VRAM usage when switching to float16+AWQ can occur because AWQ kernels are not fully optimized and may be slower than non-quantized models on some hardware ([source](https://github.com/vllm-project/vllm/issues/15105), [source](https://github.com/vllm-project/vllm/issues/21045)), and bfloat16 is not supported for AWQ quantization.

Would you like a step-by-step breakdown of why AWQ+float16 can be slower and why bfloat16 is unsupported?

Sources:

- [https://github.com/vllm-project/vllm/issues/1114](https://github.com/vllm-project/vllm/issues/1114)
- [https://github.com/vllm-project/vllm/issues/12404](https://github.com/vllm-project/vllm/issues/12404)
- [https://github.com/vllm-project/vllm/issues/11122](https://github.com/vllm-project/vllm/issues/11122)
- [https://github.com/vllm-project/vllm/issues/15105](https://github.com/vllm-project/vllm/issues/15105)
- [https://github.com/vllm-project/vllm/issues/21045](https://github.com/vllm-project/vllm/issues/21045)