@RunLLM it’s recommendated to include prompt and prompt_name as illustrated in https://ai.google.dev/gemma/docs/embeddinggemma/inference-embeddinggemma-with-sentence-transformers#using_prompts_with_embeddinggemma, how to use them in vllm?
To use prompt and prompt_name as in the EmbeddingGemma example, you must manually apply the same prompt template in vLLM, since vLLM does not natively support prompt_name or automatic prompt templating for embeddings. You should format your input string according to the template used by Sentence Transformers for the desired task (e.g., “task: search result | query: …”) and then pass this formatted string as the prompt to vLLM’s embedding API or LLM.embed() method. There is no direct prompt_name parameter in vLLM for embeddings—manual prompt construction is required for consistency with Sentence Transformers outputs, as confirmed in this vLLM issue discussion.
Would you like a step-by-step example of how to construct and use such prompts in vLLM?
Sources:
- EmbeddingGemma: Using prompts with embeddinggemma
- vLLM issue: Difference in embedding values for Alibaba-NLP/gte-Qwen2-1.5B-instruct between vLLM and huggingface methods
Hint: Mention @RunLLM in the post for followups.
1 Like