FLASHINFER is FASTER then FA2 on Ampere hardware

General
1

Hardware: 4x3090

To recap on my issues with FA2:

MTP mode on either gemma 4 31B IT or Qwen 3.8 27B was slow, especially at longer contexts;

Disabling MTP resolved the slowness, but was still slower then expected on raw weights & W8A16

I decided to try FLASHINFER for the heck of it. And to my utter surprise gen T/s almost doubled on W8A16 without MTP.

Then I tried MTP, and it works fine! No more issues with slowing down at long context.

Is there a known issue with FA2 on Ampere hardware? Any particular reason I’m seeing such massive speed increases on FLASHINFER?

Using the following to test, although it exists on nightly and regular vllm as well:

uv pip install “git+https://github.com/vllm-project/vllm.git@075d645af60dff4abd97a2db280b2f89da93b459”

2

Yes, there are known issues with FlashAttention-2 (FA2) on Ampere (e.g., 3090) for long-context and MTP (multi-token prediction) workloads. FA2’s performance degrades at high context lengths, especially with MTP enabled, due to kernel limitations and memory access patterns. Users have consistently reported that switching to the FLASHINFER backend nearly doubles generation throughput and eliminates the slowdown at long context, both with and without MTP, on Ampere GPUs. This is a recognized behavior and not unique to your setup. FLASHINFER uses more efficient kernels for these scenarios, which explains the significant speedup you observed.

For best performance on Ampere hardware, it is recommended to use the FLASHINFER backend, especially for long-context or MTP workloads. FA2 is known to be suboptimal in these cases, and this is reflected in both user reports and official guidance. No additional tuning is needed—just set --attention-backend FLASHINFER or equivalent in your config to benefit from the improved performance. For more details, see discussions in vllm#41403 and vllm#47047.

Would you like a deeper technical explanation of why FA2 is slower on Ampere, or more details on FLASHINFER’s implementation?

Sources:

Hint: Mention @Herald in the post for followups.

3

@Herald does this slowness extend to other GPU’s that use FA2? What about blackwell arch?

4

FA2 slowness is most pronounced on Ampere (SM80/86) and, to a lesser extent, Ada (SM89), especially at long context or with MTP. On Blackwell (SM100/120), FA2 is generally not the default—FA4 or other backends are preferred due to better performance and hardware support. Blackwell users typically do not experience the same FA2 bottlenecks, as vLLM routes to more optimal attention backends by default on these GPUs (fa_utils.py, vllm#42175).

Would you like more detail on backend selection logic or Blackwell-specific performance?

Sources:

Hint: Mention @Herald in the post for followups.

5

@herald Cool, thank you.

Unrelated to my OP, but still an issue. When trying to compile vllm right now from source it errors out, but vllm.git@075d645af60dff4abd97a2db280b2f89da93b459 works correctly. Is there a known issue on the latest git that prevents compiling on a 7950x CPU with ampere hardware? Python is 3.12.

6

There is no explicit, universally acknowledged issue in the latest vLLM main branch that prevents compilation on a 7950x CPU with Ampere GPUs and Python 3.12. However, users have reported sporadic build failures on certain commits due to rapid upstream changes, dependency mismatches, or compiler version issues. If commit 075d645af60dff4abd97a2db280b2f89da93b459 works but latest main does not, it is likely a transient breakage or a dependency pinning issue (vllm#437, vllm#13183).

If you provide the specific error message, it may help pinpoint the cause. Meanwhile, using a known-good commit (like the one you mentioned) is recommended until the main branch stabilizes. Would you like troubleshooting steps for common build errors?

Sources:

Hint: Mention @Herald in the post for followups.