Hello everyone, I just got access to multiple T4 GPUs to test gemma3 12b for my local users. Did someone have any experience with using this card for gemma3 before? Already checked the supported hardware and models and some github issues where people are not successful to run it. And the support f…

Running Gemma3 12B on T4 GPUs is challenging due to high VRAM requirements and numerical stability issues with float16 (the only low-precision type supported by T4). Users report that Gemma3 models often generate empty outputs or fail to run on T4, as Gemma3 is unstable with float16 and requires bfl…

Correct—T4 (compute capability 7.5, Turing architecture) does not support bfloat16, which is required for stable Gemma 3 inference. Gemma 3’s sliding window attention and numerical stability are only reliable on Ampere (compute capability 8.0+) or newer GPUs. On T4, float16 leads to empty or unstabl…

Thank you. I will try to use llama3.1 or any other older model that may fit for this hardware..

That’s a good approach. Llama 3.1 8B or similar models are known to run successfully on T4 GPUs using float16/half precision, as confirmed by multiple users ( vllm/issues/11569 , vllm/issues/1400 ). Just ensure you set dtype=“half” and adjust max_model_len and gpu_memory_utilization to avoid OOM errors…

Gemma3 on a T4 GPU

General

aarnphm May 19, 2025, 2:15pm 3

T4 is too old to run Gemma 3 unfortunately afaik.

Sliding windows attention requires at least ampere generation of GPUs here.

Topic		Replies	Views
Gemma 3 Quantization General	5	1411	June 21, 2025
Not able to run google/gemma-3n-E4B-it General	3	334	September 22, 2025
Trying to run gemma-3-27b-it-FP8-dynamic with rocm General	23	727	July 25, 2025
Which gemma-3 27B for 32GB Vram General	9	1215	September 18, 2025
Running Gemma 3 on multi-chip TPU failure Google TPU Support	5	519	May 1, 2025

Gemma3 on a T4 GPU

Related topics