# How to run a model use heterogeneous GPUs **URL:** \ **Category:** General\ **Created:** [August 12, 2025, 9:49am UTC](https://discuss.vllm.ai/t/how-to-run-a-model-use-heterogeneous-gpus/1360 "2025-08-12T09:49:57Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![longfei-zhang](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/longfei-zhang/32/593_2.png) [@longfei-zhang](https://discuss.vllm.ai/u/longfei-zhang)\ **Post date:** [August 12, 2025, 9:49am UTC](https://discuss.vllm.ai/t/how-to-run-a-model-use-heterogeneous-gpus/1360/1 "2025-08-12T09:49:57Z")
I have different GPUs，like Two Hygon DCUs: ```auto hy-smi ================================= System Management Interface ================================== ================================================================================================ HCU Temp AvgPwr Perf PwrCap VRAM% HCU% Mode 0 58.0C 135.0W auto 350.0W 85% 0.0% Normal 1 48.0C 88.0W auto 350.0W 0% 0.0% Normal ================================================================================================ ======================================== End of SMI Log ======================================== ``` Two Nvidia GPUs: ```auto NVIDIA A800 80GB PCIe ``` ```auto NVIDIA A100 80GB PCIe ``` I’m not sure how to run a model use these heterogeneous GPUs? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [August 12, 2025, 9:50am UTC](https://discuss.vllm.ai/t/how-to-run-a-model-use-heterogeneous-gpus/1360/2 "2025-08-12T09:50:12Z")
vLLM does not natively support running a single model across heterogeneous GPUs (different vendors or architectures, e.g., mixing Hygon DCUs and NVIDIA GPUs) in one inference instance. vLLM’s distributed inference (tensor/pipeline parallelism) requires all participating GPUs to be of the same type and architecture for correct operation and performance, as per [official documentation](https://docs.vllm.ai/en/latest/serving/distributed_serving.html) and [parallelism guidelines](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html). You can, however, run separate vLLM instances on each GPU type (e.g., one instance on Hygon DCUs, another on NVIDIA GPUs) and manage them independently, but you cannot shard a single model across both Hygon and NVIDIA GPUs in one vLLM process. For multi-GPU parallelism, use only identical GPUs within each vLLM instance, and select which GPUs to use via the `CUDA_VISIBLE_DEVICES` environment variable for NVIDIA cards or the equivalent for Hygon DCUs, as described in [this issue](https://github.com/vllm-project/vllm/issues/554). Would you like more detail on how to set up vLLM for each GPU type separately? Sources: - [Distributed Inference and Serving](https://docs.vllm.ai/en/latest/serving/distributed_serving.html) - [How to specify which gpu to use?](https://github.com/vllm-project/vllm/issues/554) - [parallelism guidelines](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html)