# Custom edit on the embedding out

**URL:** <https://discuss.vllm.ai/t/custom-edit-on-the-embedding-out/1725>\
**Category:** Model Support\
**Created:** [October 15, 2025, 4:27am UTC](https://discuss.vllm.ai/t/custom-edit-on-the-embedding-out/1725 "2025-10-15T04:27:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![wj210](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/wj210/32/804_2.png) [@wj210](https://discuss.vllm.ai/u/wj210)\
**Post date:** [October 15, 2025, 4:27am UTC](https://discuss.vllm.ai/t/custom-edit-on-the-embedding-out/1725/1 "2025-10-15T04:27:32Z")

</div>

Is it possible to perform an additional matrix mulitplication on the embedding outputs (post-embedding layer) on token positions that are dynamic across samples?

Furthermore, this multiplication must be performed on the output tokens during generation stage, ie each new token will have this multiplication performed on top before going through the attn/mlp layers.

I am aware that prompt embeds can be passed as inputs but this will only solve the first problem but not the second. Furthermore ,passing embeddings instead of strings would be slower?

---

<div class="post-metadata">

**Author:** ![RunLLM](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/runllm/32/233_2.png) [@RunLLM](https://discuss.vllm.ai/u/RunLLM)\
**Post date:** [October 15, 2025, 4:27am UTC](https://discuss.vllm.ai/t/custom-edit-on-the-embedding-out/1725/2 "2025-10-15T04:27:59Z")

</div>

It seems likely that vLLM does not natively support applying a custom matrix multiplication to the embedding outputs at dynamic token positions during both prompt and generation stages. While vLLM allows passing prompt embeddings directly (see [Prompt Embedding Inputs](https://docs.vllm.ai/en/latest/features/prompt_embeds.html)), this only affects the initial prompt and not the embeddings for newly generated tokens during autoregressive generation. There is no documented API or hook for modifying embeddings at each generation step before the attention/MLP layers, nor for dynamically applying a transformation at arbitrary positions per sample during generation, based on the available documentation and issues ([Prompt Embedding Inputs](https://docs.vllm.ai/en/latest/features/prompt_embeds.html), [github\_issue #20404](https://github.com/vllm-project/vllm/issues/20404)).

Regarding performance, my understanding is that passing prompt embeddings (instead of text) can be less efficient, especially for long prompts, because vLLM’s optimizations (like prefix caching and CUDA graph compilation) are designed for token IDs, not arbitrary embeddings ([Prompt Embedding Inputs](https://docs.vllm.ai/en/latest/features/prompt_embeds.html), [github\_pull\_request #24278](https://github.com/vllm-project/vllm/pull/24278)). This may result in slower inference when using embeddings as input. Would you like a step-by-step breakdown or suggestions for possible workarounds?

Sources:

- [Prompt Embedding Inputs](https://docs.vllm.ai/en/latest/features/prompt_embeds.html)
- [github\_issue #20404](https://github.com/vllm-project/vllm/issues/20404)
- [github\_pull\_request #24278](https://github.com/vllm-project/vllm/pull/24278)

Hint: Mention @RunLLM in the post for followups.

---

<div class="post-metadata">

**Author:** ![wj210](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.vllm.ai/wj210/32/804_2.png) [@wj210](https://discuss.vllm.ai/u/wj210)\
**Post date:** [October 15, 2025, 4:34am UTC](https://discuss.vllm.ai/t/custom-edit-on-the-embedding-out/1725/3 "2025-10-15T04:34:51Z")

</div>

so can i take it that it is not possible to perform this matrix multiplication for the newly generated tokens?