Anyone running DeepSeek-R1 on dedicated instances, curious if others are seeing this. Ran a structural waste analysis on Alibaba’s NSDI 2026 production data (3.54B real requests) and found 44% structural GPU waste driven purely by output token length. DeepSeek-R1 averages 2,073 output tokens vs 82 for a general 14B model each reasoning request holds the GPU 25x longer, which widens the peak/average gap structurally. The waste floor is min_waste = 1 - 1/burstiness no batching or framework tuning moves it, only pooling does. Curious if anyone has measured this on their own clusters.
Yes, this is a known structural effect with DeepSeek-R1 and similar reasoning models: their much longer average output (2,073 tokens vs ~82 for typical 14B models) causes each request to monopolize a GPU for far longer, which increases the gap between peak and average utilization. As a result, even with perfect batching, the minimum achievable waste is set by the burstiness of the workload: min_waste = 1 - 1/burstiness. No amount of batching or framework-level tuning can eliminate this waste—only pooling (i.e., sharing GPUs across multiple models or workloads) can reduce it. This phenomenon has been observed in production-scale deployments and is not unique to Alibaba’s NSDI 2026 data; it is a structural property of long-output reasoning models like DeepSeek-R1. See discussion in DeepSeek-V3 Recipe and related performance analyses.
Would you like more detail on how pooling or workload shaping can mitigate this waste?
Sources:
Hint: Mention @Herald in the post for followups.