# Gemma3 on a T4 GPU **URL:** \ **Category:** General\ **Created:** [May 19, 2025, 2:12pm UTC](https://discuss.vllm.ai/t/gemma3-on-a-t4-gpu/643 "2025-05-19T14:12:52Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![danielfl](https://avatars.discourse-cdn.com/v4/letter/d/f07891/32.png) [@danielfl](https://discuss.vllm.ai/u/danielfl)\ **Post date:** [May 19, 2025, 2:12pm UTC](https://discuss.vllm.ai/t/gemma3-on-a-t4-gpu/643/1 "2025-05-19T14:12:52Z")
Hello everyone, I just got access to multiple T4 GPUs to test gemma3 12b for my local users. Did someone have any experience with using this card for gemma3 before? Already checked the supported hardware and models and some github issues where people are not successful to run it. And the support for gemma3 gguf is still not available. But maybe someone has any trick and could help. > **[Supported Hardware — vLLM](https://docs.vllm.ai/en/v0.8.3/features/quantization/supported_hardware.html)** Thanks. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [May 19, 2025, 2:13pm UTC](https://discuss.vllm.ai/t/gemma3-on-a-t4-gpu/643/2 "2025-05-19T14:13:26Z")
Running Gemma3 12B on T4 GPUs is challenging due to high VRAM requirements and numerical stability issues with float16 (the only low-precision type supported by T4). Users report that Gemma3 models often generate empty outputs or fail to run on T4, as Gemma3 is unstable with float16 and requires bfloat16 or float32 for reliable inference—neither of which is natively supported on T4. Attempts to use float32 result in out-of-memory errors, as memory usage doubles compared to float16. There are no confirmed workarounds or tricks to make Gemma3 12B reliably work on T4 GPUs, and GGUF support for Gemma3 is still experimental and not fully available in vLLM as of now. See discussions in [vllm/issues/15097](https://github.com/vllm-project/vllm/issues/15097), [vllm/issues/16489](https://github.com/vllm-project/vllm/issues/16489), and [vllm/pull/14766](https://github.com/vllm-project/vllm/pull/14766). If you need to run Gemma3 12B, consider using GPUs with bfloat16 support (Ampere or newer) or use a smaller model variant. Would you like more detail on possible configuration tweaks or alternative models? Sources: - [vllm/issues/15097](https://github.com/vllm-project/vllm/issues/15097) - [vllm/pull/14766](https://github.com/vllm-project/vllm/pull/14766) - [vllm/issues/16489](https://github.com/vllm-project/vllm/issues/16489) ---
**Author:** ![aarnphm](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/aarnphm/32/238_2.png) [@aarnphm](https://discuss.vllm.ai/u/aarnphm)\ **Post date:** [May 19, 2025, 2:15pm UTC](https://discuss.vllm.ai/t/gemma3-on-a-t4-gpu/643/3 "2025-05-19T14:15:07Z")
T4 is too old to run Gemma 3 unfortunately afaik. Sliding windows attention requires at least ampere generation of GPUs here. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [May 19, 2025, 2:15pm UTC](https://discuss.vllm.ai/t/gemma3-on-a-t4-gpu/643/4 "2025-05-19T14:15:31Z")
Correct—T4 (compute capability 7.5, Turing architecture) does not support bfloat16, which is required for stable Gemma 3 inference. Gemma 3’s sliding window attention and numerical stability are only reliable on Ampere (compute capability 8.0+) or newer GPUs. On T4, float16 leads to empty or unstable outputs, and float32 exceeds memory limits for 12B models. See [vllm/issues/15097](https://github.com/vllm-project/vllm/issues/15097) and [vllm/issues/16489](https://github.com/vllm-project/vllm/issues/16489). Would you like more technical details or suggestions for alternative models? Sources: - [vllm/issues/15097](https://github.com/vllm-project/vllm/issues/15097) - [vllm/issues/16489](https://github.com/vllm-project/vllm/issues/16489) ---
**Author:** ![danielfl](https://avatars.discourse-cdn.com/v4/letter/d/f07891/32.png) [@danielfl](https://discuss.vllm.ai/u/danielfl)\ **Post date:** [May 19, 2025, 2:21pm UTC](https://discuss.vllm.ai/t/gemma3-on-a-t4-gpu/643/5 "2025-05-19T14:21:18Z")
Thank you. I will try to use llama3.1 or any other older model that may fit for this hardware.. ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [May 19, 2025, 2:21pm UTC](https://discuss.vllm.ai/t/gemma3-on-a-t4-gpu/643/6 "2025-05-19T14:21:55Z")
That’s a good approach. Llama 3.1 8B or similar models are known to run successfully on T4 GPUs using float16/half precision, as confirmed by multiple users ([vllm/issues/11569](https://github.com/vllm-project/vllm/issues/11569), [vllm/issues/1400](https://github.com/vllm-project/vllm/issues/1400)). Just ensure you set dtype=“half” and adjust max\_model\_len and gpu\_memory\_utilization to avoid OOM errors. Would you like a sample configuration or more tuning tips for T4? Sources: - [vllm/issues/11569](https://github.com/vllm-project/vllm/issues/11569) - [vllm/issues/1400](https://github.com/vllm-project/vllm/issues/1400)