Hello there we are using blackwell 6000 on Dell Servers and we are using openshift and k6 to do performance testing.. we have a grafana dashboard set up and using a 28B gemma model. quantized 8bit.
What we are running into is a SM Active in the high 80%s and only a SM Occupancy in the 20%ish… If we go above like 512 for our sequence limit we start seeing 80% KV Cache but our preemptions get crazy and TTFT get to about 30-40 seconds… we are using a 2g.48gb setup for the model…
we are newbies and are still trying to figure out all the nuances of the system to max the performance from our MIG setup…we know we can go to a full card however we want to learn about what the limits are and how to tweak things before throwing more hardware at it…
right now we are running K6 tests directly against vllm… throwing 500 , 1000 Virtual Users at it..
The preemptions are the clearest signal: vLLM is short of KV-cache space, and recomputing preempted requests can worsen latency. Try lowering --max-num-seqs and/or --max-num-batched-tokens; the tuning guide recommends these to reduce KV-cache pressure. Chunked prefill is enabled by default in V1 when possible: smaller max_num_batched_tokens can improve inter-token latency, while larger values favor TTFT. Sweep one setting at a time and compare preemptions, TTFT, throughput, and KV-cache use. Optimization and Tuning
For k6, test a range of concurrency levels rather than relying only on 500–1,000 virtual users: arrival rate and maximum concurrency affect what the test measures. vLLM’s benchmark guide describes unlimited-demand testing with a concurrency cap and notes that KV-cache capacity constrains sustainable concurrency. The sources don’t explain how to interpret your SM Active versus SM Occupancy metrics, or specify MIG-specific tuning limits, so I can’t diagnose those from the dashboard figures alone. Benchmarking CLI Would you like a more detailed tuning plan?