# Mixedbread reranker on vLLM \`/score\`: scores differ vs local Mixedbread; small payload = same order/different scores, large payload = different order
**URL:** https://discuss.vllm.ai/t/mixedbread-reranker-on-vllm-score-scores-differ-vs-local-mixedbread-small-payload-same-order-different-scores-large-payload-different-order/1392
**Category:** Benchmarking
**Created:** [August 15, 2025, 11:19am UTC](https://discuss.vllm.ai/t/mixedbread-reranker-on-vllm-score-scores-differ-vs-local-mixedbread-small-payload-same-order-different-scores-large-payload-different-order/1392 "2025-08-15T11:19:24Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@devang-sifthub](https://discuss.vllm.ai/u/devang-sifthub)
#### Post date: [August 15, 2025, 11:19am UTC](https://discuss.vllm.ai/t/mixedbread-reranker-on-vllm-score-scores-differ-vs-local-mixedbread-small-payload-same-order-different-scores-large-payload-different-order/1392/1 "2025-08-15T11:19:25Z")
**Mixedbread reranker on vLLM `/score`: scores differ vs local Mixedbread; small payload = same order/different scores, large payload = different order**
I’m serving Mixedbread v2 as Qwen2 seq-cls via `hf_overrides` and calling `/score`:
```bash
vllm serve mixedbread-ai/mxbai-rerank-base-v2 \
--hf_overrides '{"architectures":["Qwen2ForSequenceClassification"],"classifier_from_token":["0","1"],"method":"from_2_way_softmax"}' \
--host 0.0.0.0 --port 8000
```
Env: **AWS g6e.xlarge (L40S 48 GB)**.
Reference: **local** `from mxbai_rerank import MxbaiRerankV2` on **Mac M3 Pro (CPU)**.
Key observations:
- **Scores differ between vLLM and local Mixedbread usage.**
- **Small payload** (“To Kill a Mockingbird”): **same ranking** , but vLLM assigns **much higher probs** to negatives than Mixedbread.
- **Larger payload** (10 docs ~1K chars; query “Who wrote the novel 1984?”): **ranking order diverges** vs Mixedbread.
> **Small payload — side-by-side (vLLM vs MxbaiRerankV2)**
>
> | Doc (short) | vLLM `/score` | MxbaiRerankV2 |
> | --- | --- | --- |
> | Harper Lee (novel) | 0.9974413 | 0.9941708 |
> | Harper Lee (bio) | 0.9837090 | 0.9705565 |
> | Jane Austen | 0.9364685 | 0.2971900 |
> | Harry Potter | 0.7970418 | 0.0699584 |
> | Moby-Dick | 0.6106243 | 0.0519386 |
> | Gatsby | 0.5890393 | 0.0290230 |
> **Large payload (~1K chars/doc, “1984”) — order diverges**
>
> - Query: `Who wrote the novel 1984?`
> - 10 docs (~1K chars each). Full JSON included in the Issue ([[Usage]: vLLM `/score` with Mixedbread reranker (Qwen2 seq-cls override): \*\*scores differ vs local Mixedbread\*\*; small payload = same order/different scores; large payload (\~1K chars/doc) = \*\*order diverges\*\* · Issue #22983 · vllm-project/vllm · GitHub](https://github.com/vllm-project/vllm/issues/22983)).
**Ask:** what’s the recommended way to get **score parity** and **stable order** (label tokens, pair template, calibration) so vLLM `/score` matches the Mixedbread library?
---
### Author:  [@RunLLM](https://discuss.vllm.ai/u/RunLLM)
#### Post date: [August 15, 2025, 11:19am UTC](https://discuss.vllm.ai/t/mixedbread-reranker-on-vllm-score-scores-differ-vs-local-mixedbread-small-payload-same-order-different-scores-large-payload-different-order/1392/2 "2025-08-15T11:19:55Z")
For Mixedbread reranker models (e.g., `mixedbread-ai/mxbai-rerank-base-v2`) served via vLLM with `--hf_overrides '{"architectures":["Qwen2ForSequenceClassification"],"classifier_from_token":["0","1"],"method":"from_2_way_softmax"}'`, differences in score values and ranking order compared to the original Mixedbread library are expected, especially for longer documents. This is due to differences in prompt formatting, tokenization, and possibly numerical precision or batching behavior. For best results, you **must** use the same query/document templates as the original model—without these, vLLM scores and rankings can be unstable or incorrect. See the [official vLLM usage example](https://github.com/vllm-project/vllm/pull/19675) for the correct templates and formatting.
To maximize parity:
- Use the provided `query_template` and `document_template` to format both query and document inputs before sending to `/score`.
- Ensure tokenization and truncation settings match those of the reference implementation.
- Accept that minor score differences may persist due to hardware, precision, or batching, but ranking should be stable if templates are used.
- For large payloads, ensure that document length does not exceed model limits, and consider chunking if needed.
Would you like a step-by-step code example for formatting and querying with vLLM?
Sources:
- [vLLM PR #19675: Automatic conversion of CrossEncoding model (usage, templates, code)](https://github.com/vllm-project/vllm/pull/19675)
- [vLLM Issue #22983: User report and discussion of Mixedbread reranker score differences](https://github.com/vllm-project/vllm/issues/22983)