# Does vLLM support multiple model\_executor?

**URL:** <https://discuss.vllm.ai/t/does-vllm-support-multiple-model-executor/541>\
**Category:** Scheduling\
**Created:** [April 28, 2025, 9:53am UTC](https://discuss.vllm.ai/t/does-vllm-support-multiple-model-executor/541 "2025-04-28T09:53:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![leeeizhang](https://avatars.discourse-cdn.com/v4/letter/l/85e7bf/32.png) [@leeeizhang](https://discuss.vllm.ai/u/leeeizhang)\
**Post date:** [April 28, 2025, 9:53am UTC](https://discuss.vllm.ai/t/does-vllm-support-multiple-model-executor/541/1 "2025-04-28T09:53:14Z")

</div>

Hi, all.

My model is relatively small-scale and doesn’t require large tensor parallelism (TP) for inference (TP=2 is enough for this model). However, my hardware instance has 8 GPUs. How can I configure multiple executors to utilize the full GPU capacity while expose only single vLLM service?

Specifically, I want to group GPUs as [gpu:0,1: executor\_0], [gpu:2,3: executor\_1], etc., with a single LLM engine managing these executor groups, so it will be more easily to batching larger requests.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [April 28, 2025, 9:53am UTC](https://discuss.vllm.ai/t/does-vllm-support-multiple-model-executor/541/2 "2025-04-28T09:53:38Z")

</div>

It seems likely that you can achieve this by running multiple instances of vLLM, each using a subset of the available GPUs, and then using a load balancer to manage requests across these instances. According to [vLLM documentation](https://github.com/vllm-project/vllm/blob/main/docs/source/serving/distributed_serving.md), you can specify which GPUs to use for each instance by setting the `CUDA_VISIBLE_DEVICES` environment variable. For example, you can start one instance with `CUDA_VISIBLE_DEVICES=0,1` and another with `CUDA_VISIBLE_DEVICES=2,3`.

To expose a single vLLM service, you can use a load balancer or a proxy server to distribute incoming requests to the different vLLM instances. This setup allows you to utilize all GPUs while maintaining a single service endpoint. Would you like more detailed guidance on setting up the load balancer or proxy server?

Sources:

- [Distributed Inference and Serving](https://github.com/vllm-project/vllm/blob/main/docs/source/serving/distributed_serving.md)