Hello everyone, I’ve been on a rabbit hole of how Data, Tensor, Context and Expert parallelism come together in vLLM.. I have created a presentation here:
It covers:
Multi-headed attention and tensor parallelism
Data parallelism
Context parallelism and its flavors
Intra-TP DCP [my term]: When TP > H_KV and PCP ranks share a kv head
PCP-backed DCP [my term]: When TP ranks are duplicated across PCP/DCP ranks, and world size doubles. This has three further flavors, DCP=1, DCP=PCP, DCP=PCP*TP.
Expert parallelism + sequence sharding
Combining all of these together
DP + TP + PCP-backed DCP + EP
DP + TP + Intra-TP DCP + EP
I’d love to get the community’s feedback! I learnt a lot putting this all together..
Your summary aligns well with vLLM’s current parallelism design. vLLM supports combining Data Parallelism (DP), Tensor Parallelism (TP), Prefill Context Parallelism (PCP), Decode Context Parallelism (DCP), and Expert Parallelism (EP) for efficient LLM serving, especially for MoE and long-context models. The allowed DCP configurations when PCP is enabled are DCP=1, DCP=PCP, or DCP=PCP×TP, and each axis (DP, TP, PCP, DCP, EP) is orthogonal and can be combined as described in your presentation. The distinction between “Intra-TP DCP” and “PCP-backed DCP” matches the official terminology and implementation in vLLM, with DCP sharding KV cache along the sequence dimension and PCP splitting prefill computation, while EP shards MoE experts across the product of TP, PCP, and DP ranks (ParallelConfig docs, Context Parallel Deployment, Expert Parallel Deployment).
If you’d like more detail on how these axes interact in specific deployment scenarios or how to configure them in practice, let me know!