# Network Backend Simulation for vLLM **URL:** \ **Category:** General\ **Created:** [July 21, 2026, 6:55pm UTC](https://discuss.vllm.ai/t/network-backend-simulation-for-vllm/2812 "2026-07-21T18:55:27Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![yifeng](https://avatars.discourse-cdn.com/v4/letter/y/e99b99/32.png) [@yifeng](https://discuss.vllm.ai/u/yifeng)\ **Post date:** [July 21, 2026, 6:55pm UTC](https://discuss.vllm.ai/t/network-backend-simulation-for-vllm/2812/1 "2026-07-21T18:55:27Z")
Hello all, Please find below an RFC draft on **Network-Aware Performance Prediction for vLLM with a DAG-Based Simulation Backend**. **Motivation** : Large-scale LLM serving is often treated as a GPU scheduling problem. That is a useful abstraction inside one server. For the deployments targeted by this RFC, i.e. more than four nodes, and especially 64 nodes and above, it is incomplete. Tensor-parallel collectives, expert dispatch and combine, pipeline transfers, and KV-cache movement share a physical fabric. Their packets meet in switch queues, take different routes, and react to congestion. The network is no longer a fixed term in a latency equation; it becomes part of the serving critical path. This distinction matters because two deployment configurations, i.e. **same hardware but different NIC settings** , with the same model, GPUs, and vLLM configuration, can have very different time to first token (TTFT), time per output token (TPOT), tail latency, and sustainable throughput. A bandwidth-only model may predict the same performance for both deployments while one experiences incast, queue buildup, congestion-control oscillation, or head-of-line blocking. These transient effects are especially visible in continuous batching: a delayed collective lengthens a model step, changes the next batch assembled by the scheduler, and therefore changes the traffic that the network sees next. The tentative prototype already illustrates the missing behavior. In an eight-node, 400 Gb/s, two-tier Clos experiment, a queue-aware packet run produced message-size-dependent normalized flow completion times and DCQCN rate cuts even though no packet was dropped and no PFC event occurred. This is not yet hardware validation, but it is behavior that `bytes / bandwidth` cannot represent. Today, a credible answer usually requires the target cluster. An operator deploys a candidate configuration, generates load, collects traces, changes a parameter, and repeats. This is expensive even for an experienced infrastructure team and largely inaccessible to a newcomer who has not yet acquired or reserved the hardware. vLLM already provides realistic request generation, timed traces, arrival-rate and burstiness controls, and reports TTFT and TPOT through `vllm bench serve`; those workload semantics should be reused rather than re-created in a separate tool. What is missing is a way to execute the workload against a virtual cluster whose network has topology, queues, routing, and transport behavior. General-purpose distributed-AI simulators already demonstrate the value of graph workloads and interchangeable network models. For example, ASTRA-sim separates workload, collective, and network layers, while SimAI provides full-stack simulation and parameter studies. The gap is not another generic training simulator. The gap is a maintained, versioned connection to vLLM’s serving semantics: its V1 scheduler, continuous batching, prefill/decode transitions, parallel groups, communication backends, KV-cache state, and request-level latency metrics. This RFC proposes such a connection. A simulated serving run is represented as a sequence of dynamically generated directed acyclic graphs (DAGs). Nodes represent causally significant work—prefill and decode computation, collectives, point-to-point transfers, local reduction or copy work, and synchronization. Edges represent data, stream, ordering, and resource dependencies. Collective nodes are lowered into chunked transfers, and those transfers are executed by a packet-oriented discrete-event backend over a declared topology and fabric profile. Completion events release dependent DAG nodes. An external simulation coordinator owns virtual arrivals and completions and invokes the next supported scheduling decision at the resulting virtual time. This closed loop lets the network change subsequent batching instead of adding a fixed penalty after the fact. It requires every wall-clock dependency on the supported scheduler path to be enumerated and virtualized; unsupported paths fail validation. The simulator does not execute model tensors and does not require the target accelerator fleet. Compute durations come from explicit, versioned calibration profiles or from declared reference profiles with uncertainty bounds. Given a model and parallelism configuration, a cluster description, and a real or synthetic request distribution, the backend estimates distributions of TTFT, TPOT, throughput, link utilization, queue occupancy, and communication-induced accelerator stalls. A run with incomplete calibration is still useful for relative exploration, but it must be labeled as such and must not present an unqualified point estimate. The proposal serves two users. 1. A beginner should be able to select a model, describe a plausible workload and cluster, and receive a realistic performance envelope before obtaining the hardware. The report should explain the assumptions and identify the limiting resource, not merely return a number. 2. An on-site AI infrastructure maintainer should be able to explore a constrained configuration space before deployment. A sequential model-based search can simulate a candidate, update its search model, and propose the next candidate. The final result is a small Pareto-ranked set of the highest-ranked configurations found within the declared search space and budget, together with a bottleneck explanation. It is a planning aid; it does not apply settings to production. Simulation is complementary to tracing, not a replacement for it. Tracing explains what happened on hardware that already exists, whereas simulation can answer counterfactual questions about hardware, topology, and settings that have not been deployed. Rich profiling can also slow inference and produce large traces; vLLM’s own profiling guide explicitly warns about both effects ([profiling guide](https://docs.vllm.ai/en/stable/contributing/profiling/)). Sparse traces, microbenchmarks, and lightweight counters should calibrate and validate the simulator. Tracing remains authoritative for debugging unexpected behavior on a live system. The intended result is not a cycle-accurate digital twin and not a promise to predict every request exactly. The aim is a practical counterfactual engine that, once validated for a declared domain, can reject poor designs, rank promising configurations, estimate TTFT and TPOT envelopes, and identify whether compute, scheduling, placement, or the network is likely to dominate before an expensive deployment begins. The end-state design has four elements: 1. a versioned vLLM scheduler-step record and execution-plan-provider boundary, plus an external DAG/virtual-time contract; 2. a pluggable packet-event network backend whose end-state scope includes explicit RoCEv2, InfiniBand, and Slingshot fidelity profiles; 3. an offline simulation workflow that maps supported vLLM workload concepts to familiar serving metrics; and 4. a validation and tuning framework that reports provenance, uncertainty, and bottleneck attribution. **Proof of concept** : See the figure for a tentative run with a dense model deployed across nodes to show scale-out network activity and impact on inference overlap strategy. ![RFC_figure_0](https://canada1.discourse-cdn.com/flex036/uploads/vllm/original/2X/6/6a72a05215c9f79820bf3622210314b1510cc72d.png) Please feel free to add comments and welcome to suggestions of any kind. Cheers Yifeng Wang PhD ETH Zürich yifenwan@phys.ethz.ch