Performance Tuning for L40s with Qwen36-35B

Hardware Support NVIDIA GPU Support
1

Dear all,

I have a VM, underlined with Proxmox and 2 x Nvidia L40s. I passthrough the cards to quest VM which is Ubuntu. According to the documentation offical guide (vLLM Parallelism and Scaling), if their is no NVLINK which in my case there is no with L40s, Pipeline Parallelism is better choic then Tensor Parallelism. Also for MoE model, documentanios even suggests using Data Parallelism. I have no NUMA set on Proxmox and VM, also my PCiE is not 5 but 4 both 16x.

But when I serve my model and do a benchmark with GuideLLM, tensor parallelism is always better than DP and PP. Am I misisng something, or my benchmark not suitable to see the performance increase with PP or DP ?

My docker compose for TP is like below. For DP, I change to --tensor-parallel-size 1 --data-parallel-size 2 --enable-expert-parallel, for PP I change to --tensor-parallel-size 1 --pipeline-parallel-size 2

services:
  vllm-Qwen3.6-35B-A3B-FP8:
    image: vllm/vllm-openai:latest
    container_name: vllm-Qwen3.6-35B-A3B-FP8
    runtime: nvidia
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0,1
      # ── OpenTelemetry ──────────────────────────
      - OTEL_SERVICE_NAME=vllm-Qwen3.6-35B-A3B-FP8
      - OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/ssl/interceptor-gh.crt
      - OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE=/etc/ssl/interceptor-gh.crt
      - VLLM_LOGGING_LEVEL=DEBUG
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - ~/.cache/vllm:/root/.cache/vllm
    ports:
      - "9004:9004"
    ipc: host
    command: |
      Qwen/Qwen3.6-35B-A3B-FP8
      --gpu-memory-utilization 0.7
      --host 0.0.0.0
      --port 9004
      --tensor-parallel-size 2
      --max-model-len 256000
      --max-num-seqs 8
      --max-num-batched-tokens 16384
      --kv-cache-dtype fp8
      --enable-prefix-caching
      --api-key 123456
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --enable-chunked-prefill
      --tool-call-parser qwen3_coder
      --otlp-traces-endpoint https://otel.gh.cloud

My GuideLLM runs are :

## PP=2 Sweep

~~~bash
guidellm benchmark run \
  --target "http://localhost:9004" \
  --profile sweep \
  --rate 8 \
  --max-seconds 60 \
  --data "kind=synthetic_text,prompt_tokens=512,output_tokens=128" \
  --backend-args '{"api_key": "123456"}' \
  --output-dir ~/benchmarks \
  --outputs L40-fp8-pp-throughput-run2.json,L40-fp8-pp-throughput-run2.csv,L40-fp8-pp-throughput-run2.html
~~~


## TP=2 Sweep

~~~bash
  guidellm benchmark run \
  --target "http://localhost:9004" \
  --profile sweep \
  --rate 8 \
  --max-seconds 60 \
  --data "kind=synthetic_text,prompt_tokens=512,output_tokens=128" \
  --backend-args '{"api_key": "123456"}' \
  --output-dir ~/benchmarks \
  --outputs L40-fp8-default-throughput-run2.json,L40-fp8-default-throughput-run2.csv,L40-fp8-default-throughput-run2.html
~~~

RESULTS:

Metric TP Run PP Run
Successful Requests 58 44
Total Requests 58 44
Duration (s) 60.0 60.0
Request Latency Mean (s) 10.518 13.928
Request Latency Median (s) 10.520 13.879
Time per Output Token Mean (ms) 82.171 108.815
Time per Output Token Median (ms) 82.184 108.433
Requests/Sec Mean 0.9500 0.7167
Requests/Sec Median 0.9510 0.7195
Input Tokens Mean 5.219.310 5.219.091
Output Tokens Mean 128.0 128.0
Total Tokens Mean 6.499.310 6.499.091
Created Requests 63 49
Cancelled Requests 5 5
Queued Time Avg (s) 40.941 53.446
2

Your results are expected: on two L40s connected only by PCIe (no NVLink), tensor parallelism (TP) often outperforms pipeline parallelism (PP) and data parallelism (DP) for typical LLM inference workloads, especially with moderate batch sizes and sequence lengths. This is because, for small clusters and moderate model sizes, the communication overhead of PP and DP can outweigh their benefits, while TP’s communication is still manageable over PCIe for two GPUs. The vLLM docs state that PP is mainly for memory scaling, not guaranteed latency or throughput speedup, and is not primarily a latency optimization—especially for small clusters or small models, where inter-stage communication overhead may outweigh the benefit. DP is best for scaling throughput with many replicas, not for maximizing single-model throughput on two GPUs. For MoE models, DP+EP can help at larger scale, but with only two GPUs, TP is often still optimal for throughput and latency (vLLM Optimization Guide, vLLM-Omni Pipeline Parallelism, GitHub Issue #8089, GitHub Issue #16011).

Your benchmark is suitable: with 2 GPUs, 512 prompt tokens, and 128 output tokens, you are in a regime where TP’s communication cost is low and its memory savings allow for larger batch sizes and more KV cache, leading to better throughput and latency. PP and DP only show clear advantages when scaling to more GPUs, larger models, or when memory is the main bottleneck. For your setup, TP=2 is the recommended strategy (vLLM Optimization Guide, GitHub Issue #8089). Would you like a deeper technical breakdown of the communication patterns and scaling limits for each parallelism mode?

Sources:

Hint: Mention @Herald in the post for followups.