# Some questions about torch.compile

**URL:** <https://discuss.vllm.ai/t/some-questions-about-torch-compile/1412>\
**Category:** torch.compile integration\
**Created:** [August 17, 2025, 12:23am UTC](https://discuss.vllm.ai/t/some-questions-about-torch-compile/1412 "2025-08-17T00:23:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![wyzqwq](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/wyzqwq/32/605_2.png) [@wyzqwq](https://discuss.vllm.ai/u/wyzqwq)\
**Post date:** [August 17, 2025, 12:23am UTC](https://discuss.vllm.ai/t/some-questions-about-torch-compile/1412/1 "2025-08-17T00:23:00Z")

</div>

Hi vLLM team,

The article [“PyTorch + vLLM = ❤”](https://blog.vllm.ai/2024/06/04/pytorch-vllm.html) shows **1.05x–1.9x speedups with `torch.compile`** for models like Llama4/Qwen3/Gemma3. To better understand:

1. What **benchmark setup** (hardware, input shape) was used?

2. Was this measured by comparing **vLLM with vs. without `torch.compile` enabled**?

3. Could you share a **reference script/command** for reproducing these tests?

Thanks for your insights! This will greatly help the community validate the optimizations.

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [August 17, 2025, 12:23am UTC](https://discuss.vllm.ai/t/some-questions-about-torch-compile/1412/2 "2025-08-17T00:23:31Z")

</div>

The referenced speedups (1.05x–1.9x) for torch.compile in vLLM were measured using the vLLM benchmark suite, specifically with models like Llama3, Llama4, and Mixtral on NVIDIA H100 and AMD MI300x hardware. The benchmarks compare vLLM with torch.compile enabled (the V1 engine, where torch.compile is on by default) versus vLLM without torch.compile (V0 engine or with --enforce-eager). The [vllm-benchmark workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) is used for validation, and results are published on the [vLLM benchmark dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm) for transparency and reproducibility. Input shapes typically use batch sizes and sequence lengths relevant to LLM serving (e.g., input/output lengths of 128–256 tokens, batch sizes up to 256+), but exact shapes can be found in the benchmark scripts.

To reproduce, you can run:

```bash
python benchmarks/benchmark_throughput.py --input-len 256 --output-len 256 --model meta-llama/Meta-Llama-3-8B --num-scheduler-steps 64
# With torch.compile (V1, default)
# To disable torch.compile: add --enforce-eager

```

For more details, see the [official torch.compile integration docs](https://docs.vllm.ai/en/latest/design/torch_compile.html) and [release/benchmark instructions](https://github.com/vllm-project/vllm/blob/main/RELEASE.md). Would you like a step-by-step breakdown or more details on hardware or input shape specifics?

Sources:

- [RELEASE.md](https://github.com/vllm-project/vllm/blob/main/RELEASE.md)
- [Example benchmark command and results](https://github.com/vllm-project/vllm/pull/10984)
- [official torch.compile integration docs](https://docs.vllm.ai/en/latest/design/torch_compile.html)