What is the purpose of prompt logprobs?
The purpose of prompt_logprobs in vLLM is to return the log probabilities (logprobs) of each token in the input prompt, allowing users to evaluate how likely the model considers each prompt token, which is useful for tasks like perplexity calculation, scoring completions, or evaluating multiple-choice options without generating new tokens. This feature is especially important for LLM evaluation frameworks and tasks that require token-level likelihoods of the prompt itself, not just the generated output tokens, and is controlled via the prompt_logprobs parameter in SamplingParams during inference or generation calls, returning a dictionary of logprobs per prompt token position as part of the output structure according to vLLM documentation, API reference, and test cases.
Would you like more detail or a code example?
Sources:
Hint: Mention @RunLLM in the post for followups.
Which parts of GPU memory usage are included in the GPU utilization ratio of vLLM?
Besides the KV cache, what other parts of GPU memory usage are included in the GPU memory specified by gpu-memory-utilization?
1 Like
The model weight, activation and KV cache are included in the gpu-memory-utilization.
1 Like
Thanks for your answer!