It would be great if a request could fine tune num speculative_tokens to be fewer for thinking work and higher for coding work. On engine init specifie the max, and on requests allow a lower value such as {
“extra_body”: {
“speculative_tokens”: 4
}
}
Per-request control of num_speculative_tokens is not currently supported in vLLM; the value is set at engine initialization via the --speculative-config argument and cannot be overridden per request. The feature you describe—allowing requests to specify a lower number of speculative tokens than the engine max—is a known limitation and has been requested by users, but as of now, there is no runtime API or documented parameter to achieve this behavior in vLLM v1.x or v0.x. The only available control is at engine startup, not per request, and attempts to set per-request speculative tokens are ignored or unsupported at runtime according to Feature: Restore length-based speculation skip (speculative max_model_len) or per-request speculative decoding control in V1.
Would you like more detail on workarounds or related roadmap discussions?
Sources:
Hint: Mention @Herald in the post for followups.