I was reviewing the tpu-inference README and noticed that TPU v5e is listed as a recommended architecture. I am trying to figure out if I can run Qwen 3.5 models on a single TPU v5e-8 node.
Looking at the support matrix:
Qwen/Qwen3.5-397B-A17B is marked as fully passing (). However, since a single v5e-8 node only has about 128GB of HBM, I assume this massive model wouldn’t fit out-of-the-box. Does running this specific model strictly require a multi-host v5e cluster, or are there quantization methods available that allow it to run on a single v5e-8?
Qwen/Qwen3.5-9B is currently marked as Untested (). Has anyone in the community successfully run this smaller multimodal version on a v5e-8 yet?
Any insights on hardware requirements or configurations to get either of these running on my setup would be greatly appreciated. Thanks! @Herald
It seems likely that Qwen/Qwen3.5-397B-A17B is too large to run on a single TPU v5e-8 node (128GB HBM) without multi-host sharding, as models of this scale typically require much more memory than a single node provides. There is no explicit mention in the data sources of quantization methods (e.g., INT4/FP8) that would allow this specific 397B model to fit on a single v5e-8, nor is there confirmation of successful single-node runs for this model—multi-host clusters are generally required for such large models (see this discussion).
For Qwen/Qwen3.5-9B, there is no explicit confirmation in the data sources that it has been tested or successfully run on a v5e-8, but smaller models like 9B are much more likely to fit within the memory constraints of a single node. You may need to experiment or check with the community for up-to-date reports, as the official support matrix marks it as “Untested” (see here). Would you like more detail on quantization options or configuration tips for maximizing memory efficiency?
@Herald Thanks for clarifying that! Even if it doesn’t fit on a single v5e-8, I am still very curious about how it was successfully run on the larger v6e or v7x clusters.
Since the tpu-inference backend runs standard PyTorch model definitions, I assume the “code” used to pass the tests was a specific vLLM launch script, deployment YAML, or multi-host recipe that configures the right tensor and pipeline parallelism.
Could you point me to the specific vllm serve command, repository branch, or recipe used to officially validate the Qwen/Qwen3.5-397B-A17B model? I would love to look at the setup and see exactly how the cluster was configured to handle a model of that scale!
The data sources do not provide a specific vllm serve command, deployment YAML, or recipe used to officially validate Qwen/Qwen3.5-397B-A17B on TPU v6e or v7x clusters. There is mention of successful multi-host TPU inference for large models using Ray for distributed execution and tensor/pipeline parallelism, but no explicit command or branch is cited for this model (see here, see here). The official documentation and issues confirm that multi-host, multi-node setups with Ray and appropriate parallelism are required for such large models, but do not include a concrete example for Qwen3.5-397B-A17B.
Would you like more detail on general multi-node TPU deployment patterns with vLLM, or guidance on constructing a suitable launch command?
). However, since a single v5e-8 node only has about 128GB of HBM, I assume this massive model wouldn’t fit out-of-the-box. Does running this specific model strictly require a multi-host v5e cluster, or are there quantization methods available that allow it to run on a single v5e-8?
). Has anyone in the community successfully run this smaller multimodal version on a v5e-8 yet?