Hello all,
Please find below an RFC draft on Network-Aware Performance Prediction for vLLM with a DAG-Based Simulation Backend.
Motivation:
Large-scale LLM serving is often treated as a GPU scheduling problem. That is a useful abstraction inside one server. For the deployments targeted by this RFC, i.e. more than four nodes, and especially 64 nodes and above, it is incomplete. Tensor-parallel collectives, expert dispatch and combine, pipeline transfers, and KV-cache movement share a physical fabric. Their packets meet in switch queues, take different routes, and react to congestion. The network is no longer a fixed term in a latency equation; it becomes part of the serving critical path.
This distinction matters because two deployment configurations, i.e. same hardware but different NIC settings, with the same model, GPUs, and vLLM configuration, can have very different time to first token (TTFT), time per output token (TPOT), tail latency, and sustainable throughput. A bandwidth-only model may predict the same performance for both deployments while one experiences incast, queue buildup, congestion-control oscillation, or head-of-line blocking. These transient effects are especially visible in continuous batching: a delayed collective lengthens a model step, changes the next batch assembled by the scheduler, and therefore changes the traffic that
the network sees next.
The tentative prototype already illustrates the missing behavior. In an
eight-node, 400 Gb/s, two-tier Clos experiment, a queue-aware packet run
produced message-size-dependent normalized flow completion times and DCQCN
rate cuts even though no packet was dropped and no PFC event occurred. This is
not yet hardware validation, but it is behavior that bytes / bandwidth
cannot represent.
Today, a credible answer usually requires the target cluster. An operator
deploys a candidate configuration, generates load, collects traces, changes a
parameter, and repeats. This is expensive even for an experienced
infrastructure team and largely inaccessible to a newcomer who has not yet
acquired or reserved the hardware. vLLM already provides realistic request
generation, timed traces, arrival-rate and burstiness controls, and reports
TTFT and TPOT through vllm bench serve; those workload semantics should be
reused rather than re-created in a separate tool. What is
missing is a way to execute the workload against a virtual cluster whose
network has topology, queues, routing, and transport behavior.
General-purpose distributed-AI simulators already demonstrate the value of
graph workloads and interchangeable network models. For example, ASTRA-sim
separates workload, collective, and network layers, while SimAI provides
full-stack simulation and parameter studies. The gap is not another generic
training simulator. The gap is a maintained, versioned connection to vLLM’s
serving semantics: its V1 scheduler, continuous batching, prefill/decode
transitions, parallel groups, communication backends, KV-cache state, and
request-level latency metrics.
This RFC proposes such a connection. A simulated serving run is represented
as a sequence of dynamically generated directed acyclic graphs (DAGs). Nodes
represent causally significant work—prefill and decode computation,
collectives, point-to-point transfers, local reduction or copy work, and
synchronization. Edges represent data, stream, ordering, and resource
dependencies. Collective nodes are lowered into chunked transfers, and those
transfers are executed by a packet-oriented discrete-event backend over a
declared topology and fabric profile. Completion events release dependent DAG
nodes. An external simulation coordinator owns virtual arrivals and
completions and invokes the next supported scheduling decision at the resulting
virtual time. This closed loop lets the network change subsequent batching
instead of adding a fixed penalty after the fact. It requires every wall-clock
dependency on the supported scheduler path to be enumerated and virtualized;
unsupported paths fail validation.
The simulator does not execute model tensors and does not require the target
accelerator fleet. Compute durations come from explicit, versioned calibration
profiles or from declared reference profiles with uncertainty bounds. Given a
model and parallelism configuration, a cluster description, and a real or
synthetic request distribution, the backend estimates distributions of TTFT,
TPOT, throughput, link utilization, queue occupancy, and communication-induced
accelerator stalls. A run with incomplete calibration is still useful for
relative exploration, but it must be labeled as such and must not present an
unqualified point estimate.
The proposal serves two users.
- A beginner should be able to select a model, describe a plausible workload
and cluster, and receive a realistic performance envelope before obtaining
the hardware. The report should explain the assumptions and identify the
limiting resource, not merely return a number. - An on-site AI infrastructure maintainer should be able to explore a
constrained configuration space before deployment. A sequential
model-based search can simulate a candidate, update its search model, and
propose the next candidate. The final result is a small Pareto-ranked set of
the highest-ranked configurations found within the declared search space
and budget, together with a bottleneck explanation. It is a planning aid;
it does not apply settings to production.
Simulation is complementary to tracing, not a replacement for it. Tracing
explains what happened on hardware that already exists, whereas simulation can
answer counterfactual questions about hardware, topology, and settings that
have not been deployed. Rich profiling can also slow inference and produce
large traces; vLLM’s own profiling guide explicitly warns about both effects
(profiling guide).
Sparse traces, microbenchmarks, and lightweight counters should calibrate and
validate the simulator. Tracing remains authoritative for debugging unexpected
behavior on a live system.
The intended result is not a cycle-accurate digital twin and not a promise to
predict every request exactly. The aim is a practical counterfactual engine
that, once validated for a declared domain, can reject poor designs, rank
promising configurations, estimate TTFT and TPOT envelopes, and identify
whether compute, scheduling, placement, or the network is likely to dominate
before an expensive deployment begins.
The end-state design has four elements:
- a versioned vLLM scheduler-step record and execution-plan-provider boundary,
plus an external DAG/virtual-time contract; - a pluggable packet-event network backend whose end-state scope includes
explicit RoCEv2, InfiniBand, and Slingshot fidelity profiles; - an offline simulation workflow that maps supported vLLM workload concepts
to familiar serving metrics; and - a validation and tuning framework that reports provenance, uncertainty,
and bottleneck attribution.
Proof of concept:
See the figure for a tentative run with a dense model deployed across nodes to show scale-out network activity and impact on inference overlap strategy.
Please feel free to add comments and welcome to suggestions of any kind.
Cheers
Yifeng Wang
PhD ETH Zürich
yifenwan@phys.ethz.ch