# Speech To Text Guidance

**URL:** <https://discuss.vllm.ai/t/speech-to-text-guidance/2733>\
**Category:** General\
**Created:** [June 26, 2026, 9:05am UTC](https://discuss.vllm.ai/t/speech-to-text-guidance/2733 "2026-06-26T09:05:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ranjith](https://avatars.discourse-cdn.com/v4/letter/r/8797f3/32.png) [@Ranjith](https://discuss.vllm.ai/u/Ranjith)\
**Post date:** [June 26, 2026, 9:05am UTC](https://discuss.vllm.ai/t/speech-to-text-guidance/2733/1 "2026-06-26T09:05:48Z")

</div>

@RunLLM I am using the below docker-compose.yaml for serving transcription model. I understand how exatcly KV Cache, max-model-len would work in LLMs. But how does it work for transcription?

`services:`  
`whisper-large-v3:`  
`image: vllm/vllm-openai:v0.22.1`  
`container_name: whisper-large-v3`  
`runtime: nvidia`  
`ipc: host`  
`environment:`  
`- NVIDIA_VISIBLE_DEVICES=0`  
`volumes:`  
`- /home/ranjith/.cache/huggingface:/root/.cache/huggingface`  
`- /dev/null:/etc/ld.so.conf.d/00-cuda-compat.conf`  
`ports:`  
`- "9135:8000"`  
`command: >`  
`/root/.cache/huggingface/hub/models--openai--whisper-large-v3/snapshots/06f233fe06e710322aca913c1bc4249a0d71fce1`  
`--served-model-name whisper-large-v3`  
`--gpu-memory-utilization 0.2`  
`restart: unless-stopped`

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [June 26, 2026, 9:06am UTC](https://discuss.vllm.ai/t/speech-to-text-guidance/2733/2 "2026-06-26T09:06:13Z")

</div>

For transcription models like Whisper in vLLM, `max-model-len` and KV cache work similarly to LLMs: `max-model-len` sets the maximum number of tokens (context length) the model can process per request, and the KV cache stores intermediate activations for efficient decoding, especially during generation. However, for Whisper, the context length is typically much shorter (e.g., 448 tokens), since audio is chunked and transcribed in segments, so the KV cache and `max-model-len` mainly affect how long an audio segment can be processed at once and how much GPU memory is reserved for concurrent requests, not the total audio file length or transcription quality. See [docs: Conserving Memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory/) and [Speech-to-Text API](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/#transcriptions-api) for details.

Would you like a step-by-step explanation of how these parameters impact Whisper transcription throughput and memory usage?

Sources:

- [docs: Conserving Memory](https://docs.vllm.ai/en/latest/configuration/conserving_memory/)

Hint: Mention @Herald in the post for followups.