# Question about profile run **URL:** \ **Category:** General\ **Created:** [July 26, 2025, 7:43am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194 "2025-07-26T07:43:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Infinite666](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/infinite666/32/531_2.png) [@Infinite666](https://discuss.vllm.ai/u/Infinite666)\ **Post date:** [July 26, 2025, 7:43am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/1 "2025-07-26T07:43:28Z")
I have some questions about profile run. As I known, the `profile_run` performs a naive inference with `max_num_batched_tokens` tokens to get the peak memory during inference. I wonder why doesn’t the `profile_run` need to consider about the possible computed tokens? If chunked prefill is enabled, there input may be like [computed\_tokens + `max_num_batched_tokens` of new tokens]. Let’s say `N=max_num_batched_tokens` and we have a request contains 2\*N input tokens. The first chunked prefill will prefill for the first N tokens, then the second chunked prefill will prefill for the last N tokens while having N computed tokens. ![image](https://canada1.discourse-cdn.com/flex036/uploads/vllm/original/1X/ea17be34255ad4bbff0cfe713a5c59b7271ea34a.png) So, it seems like `profile_run` will only run for the circumstances like the First Chunked Prefill, but isn’t the Second Chunked Prefill will use more memory because of the longer sequence length? If so, if we have small `max_num_batched_tokens` and big `max_model_len` when enabled chunked prefill, vLLM maybe OOM? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [July 26, 2025, 7:43am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/2 "2025-07-26T07:43:56Z")
You are correct: when chunked prefill is enabled, the second (and later) chunked prefill steps process both the previously computed tokens (up to `max_model_len`) and a new chunk of `max_num_batched_tokens` tokens. However, the `profile_run` in vLLM only simulates a single batch of `max_num_batched_tokens` tokens, not the scenario where a sequence has both a long prefix of computed tokens and a new chunk, which could underestimate peak memory usage in some cases. This means that if `max_model_len` is much larger than `max_num_batched_tokens`, and chunked prefill is enabled, there is a risk of OOM during actual inference, since the memory required for the full sequence (computed + new tokens) may exceed what was profiled and reserved during `profile_run` ([source](https://github.com/vllm-project/vllm/issues/6706051), [source](https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py)). This limitation is acknowledged in the community, and the current implementation does not fully account for the memory usage of long sequences with chunked prefill. As a result, users should be cautious when setting a small `max_num_batched_tokens` and a large `max_model_len` together, as this can indeed lead to OOM errors in practice ([source](https://github.com/vllm-project/vllm/issues/6706051)). Would you like a step-by-step breakdown of how memory usage can exceed the profiled amount in this scenario? Sources: - [https://github.com/vllm-project/vllm/blob/main/vllm/worker/model\_runner.py](https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py) ---
**Author:** ![Infinite666](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/infinite666/32/531_2.png) [@Infinite666](https://discuss.vllm.ai/u/Infinite666)\ **Post date:** [July 26, 2025, 7:52am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/3 "2025-07-26T07:52:35Z")
Could you show me the former discussion of this limitation in the community? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [July 26, 2025, 7:52am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/4 "2025-07-26T07:52:59Z")
Yes. This limitation has been discussed in the vLLM community. For example, in [this GitHub issue](https://github.com/vllm-project/vllm/issues/8185), a user observed that the number of allocated GPU blocks (KV cache) depends on `max_model_len`, and the explanation was that during `profile_run`, vLLM profiles memory usage with `max_num_batched_tokens` (defaulting to `max_model_len`) to ensure at least one sequence of `max_model_len` tokens can be processed. This means that if `max_model_len` is much larger than `max_num_batched_tokens`, the actual memory usage during chunked prefill with long sequences may exceed what was profiled, potentially causing OOM. Additionally, in [this issue](https://github.com/vllm-project/vllm/issues/5533), it was noted that chunked prefill and prefix caching together can lead to unexpected memory usage, and the interaction between these features and memory profiling is still being improved. Would you like more details or direct quotes from these discussions? Sources: - [https://github.com/vllm-project/vllm/issues/8185](https://github.com/vllm-project/vllm/issues/8185) - [https://github.com/vllm-project/vllm/issues/5533](https://github.com/vllm-project/vllm/issues/5533) ---
**Author:** ![Infinite666](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/infinite666/32/531_2.png) [@Infinite666](https://discuss.vllm.ai/u/Infinite666)\ **Post date:** [July 26, 2025, 8:07am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/5 "2025-07-26T08:07:13Z")
I don’t think these issues are relative to my questions. So what’s the plan of vllm to deal with this limitation that may cause OOM? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [July 26, 2025, 8:07am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/6 "2025-07-26T08:07:28Z")
Currently, vLLM does not have a built-in mechanism to fully prevent OOM in this scenario; users must manually set parameters like `max_model_len` and `max_num_batched_tokens` to avoid OOM ([source](https://github.com/vllm-project/vllm/issues/5907)). There is ongoing discussion and design work to address this, such as proposals for “chunked logits processing” and improvements to memory profiling, but no finalized solution has been merged yet ([source](https://github.com/vllm-project/vllm/issues/5907)). For now, the recommended workaround is to use `--enable-chunked-prefill` and set `--max_num_batched_tokens` to match `--max_num_seqs`, which can help avoid OOM but may reduce prefill speed. The vLLM team acknowledges this as a significant issue and plans to revisit it after current performance optimizations are complete. Sources: - [https://github.com/vllm-project/vllm/issues/5907](https://github.com/vllm-project/vllm/issues/5907) ---
**Author:** ![Infinite666](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/infinite666/32/531_2.png) [@Infinite666](https://discuss.vllm.ai/u/Infinite666)\ **Post date:** [July 26, 2025, 8:15am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/7 "2025-07-26T08:15:32Z")
I still don’t think the source above is relate to my questions, do you have any roadmap or milestone that vllm team has to deal with this? ---
**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\ **Post date:** [July 26, 2025, 8:15am UTC](https://discuss.vllm.ai/t/question-about-profile-run/1194/8 "2025-07-26T08:15:48Z")
There is no explicit roadmap item or milestone in the official vLLM roadmaps that directly addresses improving `profile_run` to account for the combined memory usage of computed tokens and new tokens during chunked prefill, or to fully prevent OOM in this scenario. The current roadmaps focus on performance optimizations, chunked prefill (which is now always enabled), and memory management improvements, but do not mention a specific plan to fix this profiling limitation ([Q1 2025 Roadmap](https://github.com/vllm-project/vllm/issues/11862), [Q4 2024 Roadmap](https://github.com/vllm-project/vllm/issues/9006)). If this limitation is critical for your use case, the vLLM team encourages users to open a feature request or comment on the roadmap issues to raise its priority. Sources: - [https://github.com/vllm-project/vllm/issues/11862](https://github.com/vllm-project/vllm/issues/11862) - [https://github.com/vllm-project/vllm/issues/9006](https://github.com/vllm-project/vllm/issues/9006)