# Running Gemma 3 on multi-chip TPU failure

**URL:** <https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377>\
**Category:** Google TPU Support\
**Created:** [April 4, 2025, 8:44pm UTC](https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377 "2025-04-04T20:44:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vanbasten23](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/vanbasten23/32/165_2.png) [@vanbasten23](https://discuss.vllm.ai/u/vanbasten23)\
**Post date:** [April 4, 2025, 8:44pm UTC](https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377/1 "2025-04-04T20:44:27Z")

</div>

Hi.

I’m running Gemma3 27b on multi-chip TPU via

```auto
MODEL=google/gemma-3-27b-it
VLLM_USE_V1=1 vllm serve $MODEL --seed 42 --disable-log-requests --gpu-memory-utilization 0.95 --max-num-batched-tokens 4096 --max-num-seqs 512 --tensor-parallel-size 4 --max-model-len 2048 &
python benchmarks/benchmark_serving.py --backend vllm --model $MODEL --dataset-name random --random-input-len 1800 --random-output-len 128 --random-prefix-len 0 --num-prompts 1000

```

The server was up successfully but when I send the benchmarking request, it crashes with an error:

```auto
INFO: Application startup complete.
python benchmarks/benchmark_serving.py --backend vllm --mopython benchmarks/benchmark_serving.py --backend vllm --model $MODEL --dataset-name random --random-input-len 1800 --random-output-len 128 --random-prefix-len 0 --num-prompts 1000
INFO 04-04 20:14:52 [loggers.py:87] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
WARNING:root:libtpu.so and TPU device found. Setting PJRT_DEVICE=TPU.
INFO 04-04 20:14:53 [__init__.py:239] Automatically detected platform tpu.
Namespace(backend='vllm', base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/completions', dataset_name='random', dataset_path=None, max_concurrency=None, model='google/gemma-3-27b-it', tokenizer=None, use_beam_search=False, num_prompts=1000, logprobs=None, request_rate=inf, burstiness=1.0, seed=0, trust_remote_code=False, disable_tqdm=False, profile=False, save_result=False, save_detailed=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='ttft,tpot,itl', metric_percentiles='99', goodput=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, random_input_len=1800, random_output_len=128, random_range_ratio=1.0, random_prefix_len=0, hf_subset=None, hf_split=None, hf_output_len=None, tokenizer_mode='auto', served_model_name=None, lora_modules=None)
INFO 04-04 20:15:02 [loggers.py:87] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
Starting initial single prompt test run...
INFO: 127.0.0.1:41240 - "POST /v1/completions HTTP/1.1" 200 OK
CRITICAL 04-04 20:15:09 [multiproc_executor.py:49] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
CRITICAL 04-04 20:15:09 [core_client.py:361] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
Traceback (most recent call last):
  File "/mnt/disks/persist/vllm/benchmarks/benchmark_serving.py", line 1030, in <module>
    main(args)
  File "/mnt/disks/persist/vllm/benchmarks/benchmark_serving.py", line 659, in main
    benchmark_result = asyncio.run(
  File "/home/xiowei/miniconda3/envs/vllm/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/xiowei/miniconda3/envs/vllm/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/mnt/disks/persist/vllm/benchmarks/benchmark_serving.py", line 294, in benchmark
    raise ValueError(
ValueError: Initial test run failed - Please make sure benchmark arguments are correctly specified. Error: Traceback (most recent call last):
  File "/home/xiowei/miniconda3/envs/vllm/lib/python3.10/site-packages/aiohttp/client_proto.py", line 93, in connection_lost
    uncompleted = self._parser.feed_eof()
  File "aiohttp/_http_parser.pyx", line 508, in aiohttp._http_parser.HttpParser.feed_eof
aiohttp.http_exceptions.TransferEncodingError: 400, message:
  Not enough data for satisfy transfer length header.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/disks/persist/vllm/benchmarks/backend_request_func.py", line 287, in async_request_openai_completions
    async for chunk_bytes in response.content:
  File "/home/xiowei/miniconda3/envs/vllm/lib/python3.10/site-packages/aiohttp/streams.py", line 52, in __anext__
    rv = await self.read_func()
  File "/home/xiowei/miniconda3/envs/vllm/lib/python3.10/site-packages/aiohttp/streams.py", line 352, in readline
    return await self.readuntil()
  File "/home/xiowei/miniconda3/envs/vllm/lib/python3.10/site-packages/aiohttp/streams.py", line 386, in readuntil
    await self._wait("readuntil")
  File "/home/xiowei/miniconda3/envs/vllm/lib/python3.10/site-packages/aiohttp/streams.py", line 347, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <TransferEncodingError: 400, message='Not enough data for satisfy transfer length header.'>

```

From the error, it’s not very clear about where the real failure is. Has anyone seen something similar? Thanks!

---

<div class="post-metadata">

**Author:** ![vanbasten23](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/vanbasten23/32/165_2.png) [@vanbasten23](https://discuss.vllm.ai/u/vanbasten23)\
**Post date:** [April 7, 2025, 6:33pm UTC](https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377/2 "2025-04-07T18:33:31Z")

</div>

I tried the offline test by running a modified [tpu.py](https://gist.github.com/vanbasten23/56ce17112813ef2d60738a9e11ab353e), and it fails too: [gist:d641f2c1e33418f48d27cfc65386937f · GitHub](https://gist.github.com/vanbasten23/d641f2c1e33418f48d27cfc65386937f)

Though, the error `MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.` is very vague and doesn’t say exactly what fails. Also, the precompilation succeeded as you can see `Compilation finished in in 4.54`.

Note gemma3 1b on single chip works.

---

<div class="post-metadata">

**Author:** ![vanbasten23](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/vanbasten23/32/165_2.png) [@vanbasten23](https://discuss.vllm.ai/u/vanbasten23)\
**Post date:** [April 22, 2025, 2:24am UTC](https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377/3 "2025-04-22T02:24:12Z")

</div>

I pulled the latest code from main and I was able to see the real error: [gist:dad1febba89027747cb740b1e042e14a · GitHub](https://gist.github.com/vanbasten23/dad1febba89027747cb740b1e042e14a) by running the tpu.py [script](https://gist.github.com/vanbasten23/56ce17112813ef2d60738a9e11ab353e).

I used a smaller model `google/gemma-3-1b-it` but it didn’t trigger the error. I think it’s because gemma-3-1b-it does not support multi-modality.  
But `google/gemma-3-4b-it` and `google/gemma-3-27b-it` result in the same error.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [April 22, 2025, 2:24am UTC](https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377/4 "2025-04-22T02:24:39Z")

</div>

The error you are encountering seems to be related to the multi-modality support in the Gemma 3 models. According to the [vLLM documentation](https://docs.vllm.ai/en/stable/models/supported_models.html#id3), the Gemma 3 models, such as `google/gemma-3-4b-it` and `google/gemma-3-27b-it`, support text and image inputs. However, the `google/gemma-3-1b-it` model is likely text-only, which is why it does not trigger the same error.

If you are experiencing issues with multi-modality, it might be due to the current limitations in vLLM’s handling of mixed attention patterns for text and image inputs. The documentation notes that V0 correctly implements the model’s attention pattern, but V1 uses a simplified pattern that may not fully support the original model’s behavior, especially with image inputs. This could be causing the error you are seeing with the larger models that support multi-modality.

Would you like more details on how to potentially resolve this issue or further information on the differences between V0 and V1 handling of these models?

Sources:

- [vLLM documentation](https://docs.vllm.ai/en/stable/models/supported_models.html#id3)

---

<div class="post-metadata">

**Author:** ![vanbasten23](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/vanbasten23/32/165_2.png) [@vanbasten23](https://discuss.vllm.ai/u/vanbasten23)\
**Post date:** [May 1, 2025, 5:05pm UTC](https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377/5 "2025-05-01T17:05:28Z")

</div>

I figured out the issue. The issue is that when we use torch.compile, it is using an incorrect backend (inductor) on TPU. That’s why I got the aforementioned error.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [May 1, 2025, 5:05pm UTC](https://discuss.vllm.ai/t/running-gemma-3-on-multi-chip-tpu-failure/377/6 "2025-05-01T17:05:57Z")

</div>

You’re correct: on TPU, vLLM should use the “openxla” backend with `torch.compile`, not “inductor”. Using “inductor” on TPU is unsupported and will cause errors, as seen in your logs. The vLLM TPU integration explicitly sets the backend to “openxla” for TPU execution, and this is enforced in the code and config logic ([tpu.py](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/tpu.py), [tpu\_model\_runner.py](https://github.com/vllm-project/vllm/blob/main/vllm/worker/tpu_model_runner.py)).

If you see “inductor” being used on TPU, it likely means the backend is not being set correctly in your environment or code. Make sure your vLLM config and environment variables do not override the backend, and that you are running a recent version of vLLM with proper TPU support ([Initial TPU integration PR](https://github.com/vllm-project/vllm/pull/5292)).

Would you like step-by-step guidance on how to ensure the correct backend is set for TPU in your setup?

Sources:

- [tpu.py](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/tpu.py)
- [tpu\_model\_runner.py](https://github.com/vllm-project/vllm/blob/main/vllm/worker/tpu_model_runner.py)
- [Initial TPU integration PR](https://github.com/vllm-project/vllm/pull/5292)