@RunLLM I am using the below docker-compose.yaml for serving transcription model. I understand how exatcly KV Cache, max-model-len would work in LLMs. But how does it work for transcription?
For transcription models like Whisper in vLLM, max-model-len and KV cache work similarly to LLMs: max-model-len sets the maximum number of tokens (context length) the model can process per request, and the KV cache stores intermediate activations for efficient decoding, especially during generation. However, for Whisper, the context length is typically much shorter (e.g., 448 tokens), since audio is chunked and transcribed in segments, so the KV cache and max-model-len mainly affect how long an audio segment can be processed at once and how much GPU memory is reserved for concurrent requests, not the total audio file length or transcription quality. See docs: Conserving Memory and Speech-to-Text API for details.
Would you like a step-by-step explanation of how these parameters impact Whisper transcription throughput and memory usage?