Pick the right embedding model: build your own retrieval benchmark

What to get right before you pick an embedding model

• MTEB's overall average blends eight unrelated tasks — clustering, classification, bitext mining, summarisation and more, alongside retrieval. A model can rank near the top of the leaderboard and still sit well behind the field on retrieval specifically, which is the only task most RAG systems care about.
• Shortlist on paper first — dimensions, maximum sequence length, licence and price per token narrow a field of hundreds down to five or six realistic candidates before you spend a rupee or a pound on evaluation compute.
• Build a small golden set from your own corpus — a few hundred query-document pairs, generated with an LLM and spot-checked by a human, beats any public benchmark at predicting how a model will behave on your traffic.
• Recall@k, MRR and NDCG each tell you something different. Recall@k says whether the right document made the cut; MRR and NDCG say how well it was ranked once it did.
• Dimension count is a cost decision, not just a quality one. Storage scales roughly linearly with dimensions, so a 3,072-dimension model can double your vector database bill against a 1,536-dimension one for a retrieval gain you have not yet measured.
• Re-run the whole harness on every model upgrade. A "drop-in" replacement from your provider can move recall by several points on your specific queries without ever showing up as a headline change.

None of this is exotic engineering. It is closer to a lab discipline: define what "good" means for your documents, measure it, and keep the receipts every time something changes. Skipping it is how teams end up defending a model choice with a screenshot of a leaderboard instead of a number they can reproduce.

Why MTEB's average misleads you on retrieval

The Massive Text Embedding Benchmark is the default starting point for almost every embedding model comparison published in 2026, and it earns that position — it covers dozens of public datasets across a genuinely wide range of tasks and languages, and every serious lab now reports against it. The problem is not that MTEB is a bad benchmark. The problem is what happens when you collapse it into a single number and shop by that number alone.

The task-mix problem

MTEB's English leaderboard spans eight distinct task categories — bitext mining, classification, clustering, pair classification, reranking, retrieval, semantic textual similarity and summarisation — across roughly sixty datasets, each scored with its own metric. Retrieval specifically is scored with NDCG@10. A model's overall MTEB average is essentially an unweighted blend of all eight, which means a model that is exceptional at clustering customer-support tickets or classifying sentiment can pull its average well above a model that is narrowly, specifically better at retrieval. If your application is RAG, you do not care about seven of those eight tasks. You care about one. Shopping by the blended average is a bit like hiring a generalist athlete for a sprint final because their decathlon score was higher — the number is real, it is just answering a different question than the one you are asking.

The overfitting problem

The second issue is that MTEB's datasets are public, and most of them have been public for years. Model providers self-report their scores, and because the underlying corpora are freely downloadable, a model trained or fine-tuned after a dataset's release may have seen it — or close paraphrases of it — during pretraining, which quietly inflates the reported number in a way the leaderboard has no mechanism to catch. This is sometimes called within-benchmark overfitting: a model posts an outsized win on a specific MTEB dataset yet underperforms on a held-out corpus with a similar shape, because it learned the benchmark rather than the underlying task. Independent analyses of national-language MTEB leaderboards have gone further and found that the score gaps between several top-ranked models are not statistically significant once you account for the natural variance between individual test items — several supposedly "different" models are, in practice, tied.

None of this means MTEB is useless — quite the opposite. It is an excellent, fast way to go from "every embedding model that exists" to "five or six models worth taking seriously." The mistake is stopping there. Read the retrieval-specific column, not the overall average, use it to build a shortlist, and then verify that shortlist on your own documents before anything reaches production.

Shortlist candidates on the criteria that actually matter

Before you embed a single document, narrow your options on paper. Five properties do most of the filtering work: native dimension count (and whether it supports Matryoshka truncation), maximum input sequence length, licence terms, indicative price, and whether the model is an API call or a self-hosted download. The table below sketches the shape of the decision using well-known models across each category as of July 2026 — treat the exact figures as illustrative; prices and licence terms move fast enough that you should re-verify against the vendor's own page before committing budget.

Model	Native dims (Matryoshka range)	Max input length	Licence / hosting	Indicative price
OpenAI text-embedding-3-large	3,072 (1,024 / 256)	8,191 tokens	Proprietary API	$0.13 / 1M tokens
OpenAI text-embedding-3-small	1,536	8,191 tokens	Proprietary API	$0.02 / 1M tokens
Voyage voyage-3.5	2,048 (1,024 / 512 / 256) + int8/binary	32,000 tokens	Proprietary API	$0.06 / 1M tokens
Cohere Embed v4	1,536 (256 / 512 / 1,024), multimodal	128,000 tokens	Proprietary API	$0.12 / 1M text tokens
BGE-M3 (BAAI)	1,024 dense (+ sparse, multi-vector)	8,192 tokens	MIT — self-host	Free weights; compute only
gte-Qwen2-7B-instruct (Alibaba)	3,584	32,000 tokens	Apache 2.0 — self-host	Free weights; compute only
Qwen3-Embedding (0.6B / 4B / 8B)	1,024–4,096 (MRL-flexible)	32,768 tokens	Apache 2.0 — self-host	Free weights; compute only
Jina embeddings v4	2,048 (down to 128)	Long-context, multimodal	Restrictive research licence — verify current terms	Self-host, licence-gated

Two rows deserve a second look before you file them as "safe defaults." Jina's v4 weights carry licence terms that have shifted between releases and have historically restricted commercial use without a separate agreement — attractive dimensions and multilingual reach are worth nothing if legal will not sign off, so confirm the exact licence attached to the specific checkpoint you plan to ship, not the one referenced in a six-month-old blog post. And every open-weight row in that table trades a $0 licence fee for a GPU bill and an operations burden — hosting your own embedding model is a real infrastructure decision, not a free lunch, and it deserves the same scrutiny as any other self-host-versus-API call.

Sequence length matters more than it looks on a spec sheet. If your chunking strategy produces 500-token passages, an 8,191-token ceiling is irrelevant. If you embed whole support tickets, long contracts, or entire meeting transcripts as single units — a pattern we cover in our chunking and embedding strategies guide — a model capped at 8,000 tokens will silently truncate your input, and truncation is a retrieval bug that looks exactly like a bad embedding model until you check the input length.

Build a golden set of query-document pairs from your own corpus

A public benchmark tells you how a model performs on someone else's questions about someone else's documents. What you actually need to know is how it performs on your questions about your documents — your product terminology, your document structure, your two markets' spelling and phrasing. The fix is a golden set: a modest collection of (query, relevant document) pairs pulled from your real corpus.

Hand-writing hundreds of realistic queries is slow, so the practical approach is to sample chunks from your corpus and ask an LLM to generate plausible user queries for each one, then have a human spot-check a sample of the output before it becomes your ground truth. This keeps the cost down while still producing pairs that reflect your actual documents rather than a public dataset's.

import json
from openai import OpenAI

client = OpenAI()

PROMPT_TEMPLATE = """You are helping build a retrieval evaluation set.
Read the passage below and write {n} short, realistic user queries that
this passage — and only this passage — would be the best answer to.
Avoid queries so generic that many other passages in a large knowledge
base could answer them equally well. Return a JSON array of strings,
nothing else.

Passage:
\"\"\"{passage}\"\"\"
"""


def generate_queries(passage: str, n: int = 3, model: str = "gpt-5.1-mini") -> list:
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": PROMPT_TEMPLATE.format(n=n, passage=passage)}],
        temperature=0.7,
    )
    raw = resp.choices[0].message.content
    try:
        queries = json.loads(raw)
    except json.JSONDecodeError:
        return []
    return [q.strip() for q in queries if isinstance(q, str) and q.strip()]


def build_golden_set(chunks: dict, queries_per_chunk: int = 3) -> list:
    """chunks: doc_id -> passage text, sampled from your own corpus."""
    golden_set = []
    for doc_id, passage in chunks.items():
        for query in generate_queries(passage, n=queries_per_chunk):
            golden_set.append({
                "query_id": f"{doc_id}-{len(golden_set)}",
                "query_text": query,
                "relevant_ids": [doc_id],   # single-passage ground truth
            })
    return golden_set

Two details make the difference between a golden set that predicts production behaviour and one that flatters whichever model you already liked. First, sample chunks proportionally to your real traffic mix — if a third of your corpus is product documentation and a tenth is legal terms, your golden set should roughly reflect that split, not an even distribution across document types. Second, never skip the human pass: read a random 10 to 15 percent of the generated queries against their source passage and drop or rewrite anything ambiguous, anything that could equally be answered by a different document, or anything the LLM generated by lifting a sentence verbatim rather than genuinely paraphrasing intent. A verbatim-lifted query makes retrieval trivially easy and will flatter every candidate model equally, which defeats the entire point of the exercise.

If your benchmark later shows every off-the-shelf candidate falling short on your domain vocabulary — regulatory language, product SKUs, a regional dialect — the next lever is not a bigger model but a fine-tuned one. Our guide to fine-tuning embedding and reranker models for domain RAG picks up exactly where this one leaves off, using the same golden-set discipline to decide whether fine-tuning is worth the effort.

Run the benchmark: Recall@k, MRR and NDCG in Python

With a golden set and a shortlist of candidate models, the mechanics are straightforward: embed every document chunk and every golden-set query with a candidate model, retrieve the top results for each query by cosine similarity, and score the ranked list against the known relevant document. Three metrics cover what matters. Recall@k asks whether the relevant document made it into the top k results at all — it is rank-insensitive, so a hit at position one and a hit at position ten score identically. MRR (Mean Reciprocal Rank) rewards getting the first relevant result as close to position one as possible, which matters when your application reads or displays only the top hit. NDCG (Normalised Discounted Cumulative Gain) folds in both position and graded relevance, discounting a relevant result logarithmically the further down the ranked list it sits.

For a benchmark run over a few hundred golden-set queries against a corpus of a few hundred thousand chunks, brute-force cosine similarity over a NumPy matrix is fast enough and avoids introducing an approximate-nearest-neighbour index as a confounding variable — you are testing embedding quality, not your production vector database's recall behaviour, which is a separate question our vector database selection guide covers.

import numpy as np


def top_k_cosine(query_vec, doc_matrix, doc_ids, k):
    """doc_matrix: (n_docs, dim), L2-normalised rows.
    query_vec: (dim,), L2-normalised."""
    scores = doc_matrix @ query_vec
    top_idx = np.argsort(-scores)[:k]
    return [doc_ids[i] for i in top_idx]


def recall_at_k(retrieved_ids, relevant_ids, k):
    if not relevant_ids:
        return None
    hits = len(set(retrieved_ids[:k]) & set(relevant_ids))
    return hits / len(relevant_ids)


def reciprocal_rank(retrieved_ids, relevant_ids):
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / rank
    return 0.0


def dcg_at_k(retrieved_ids, relevance_by_id, k):
    dcg = 0.0
    for rank, doc_id in enumerate(retrieved_ids[:k], start=1):
        rel = relevance_by_id.get(doc_id, 0)
        if rel:
            dcg += (2 ** rel - 1) / np.log2(rank + 1)
    return dcg


def ndcg_at_k(retrieved_ids, relevance_by_id, k):
    dcg = dcg_at_k(retrieved_ids, relevance_by_id, k)
    ideal = sorted(relevance_by_id.values(), reverse=True)[:k]
    idcg = sum((2 ** rel - 1) / np.log2(rank + 1) for rank, rel in enumerate(ideal, start=1))
    return dcg / idcg if idcg > 0 else 0.0


def run_benchmark(golden_set, query_vecs, doc_matrix, doc_ids, k_values=(5, 10)):
    """golden_set: list of {"query_id", "relevant_ids"} dicts.
    query_vecs: dict of query_id -> L2-normalised embedding."""
    max_k = max(max(k_values), 10)
    results = {f"recall@{k}": [] for k in k_values}
    results["mrr"] = []
    results["ndcg@10"] = []

    for item in golden_set:
        qid = item["query_id"]
        relevant = set(item["relevant_ids"])
        retrieved = top_k_cosine(query_vecs[qid], doc_matrix, doc_ids, max_k)

        for k in k_values:
            r = recall_at_k(retrieved, relevant, k)
            if r is not None:
                results[f"recall@{k}"].append(r)

        results["mrr"].append(reciprocal_rank(retrieved, relevant))

        relevance_by_id = {doc_id: 1 for doc_id in relevant}
        results["ndcg@10"].append(ndcg_at_k(retrieved, relevance_by_id, 10))

    return {metric: float(np.mean(vals)) for metric, vals in results.items() if vals}

Run this once per candidate model over the same golden set and you get a small table of comparable numbers — Recall@5, Recall@10, MRR and NDCG@10 for each. Read them together, not in isolation. A model with the highest Recall@10 but a mediocre MRR is finding the right document but burying it near the bottom of the results your reranker or prompt will see; a model with strong MRR but weak Recall@10 is excellent when it works and silently misses the right document more often than the other candidate. This benchmark measures retrieval only — it says nothing about whether your generator uses the retrieved context faithfully, which is a separate, complementary question our RAGAS evaluation guide covers in depth. Run both: this harness picks the embedding model, RAGAS tells you whether the whole pipeline is trustworthy once it is wired up.

Weigh the trade-offs: cost, latency, storage and Matryoshka dimensions

Once your benchmark produces a ranked shortlist, the winner on quality alone is not automatically the model you should ship. Dimension count drives storage cost close to linearly, and storage is not the only line item it touches — larger vectors also mean more memory for your approximate-nearest-neighbour index and, typically, slower query latency at scale.

Dimensions	Bytes per vector (float32)	Storage for 10M vectors	Relative storage cost
256	1 KB	~10 GB	1×
768	3 KB	~30 GB	3×
1,024	4 KB	~40 GB	4×
1,536	6 KB	~60 GB	6×
3,072	12 KB	~120 GB	12×

Those figures are raw float32 vector storage only; most approximate-nearest-neighbour indexes such as HNSW add roughly another fifth to a half on top in graph overhead, and quantisation — int8 or binary, offered by several of the API vendors in the earlier table — can claw a large share of that back at a small, measurable cost to recall. Treat the table as a floor, and re-run your Recall@k, MRR and NDCG harness on the quantised or truncated vectors before you assume the saving is free.

Matryoshka embeddings: fewer dimensions, most of the signal

Several models in the earlier shortlist table — OpenAI's text-embedding-3 family, Voyage's newer models, Cohere Embed v4, and Qwen3-Embedding among them — are trained with Matryoshka Representation Learning (MRL). An MRL-trained model is optimised so that its earliest dimensions carry the most important signal, meaning you can truncate the output vector to a fraction of its native size and keep the great majority of retrieval quality, without retraining anything. OpenAI's own comparison found a text-embedding-3-large vector truncated to 256 dimensions still outperformed the older ada-002 model at its full 1,536 dimensions on MTEB, and a widely cited Sentence Transformers demonstration found a Matryoshka-trained model kept roughly 98 percent of its full performance at about an eighth of its original dimension count — while a conventionally trained model truncated to the same size lost noticeably more.

That gap between "trained for truncation" and "truncated anyway" is the detail worth remembering: MRL is a specific training objective, not a property every embedding model has by default. Truncating a model that was never trained this way is a cruder operation and the quality falls off faster. Before you dial dimensions down to save storage, confirm your chosen model documents Matryoshka training, and then verify — on your own golden set, with the Recall@k, MRR and NDCG harness above — exactly how much quality you keep at each dimension size you are considering. A vendor's headline retention number was measured on their benchmark, not yours.

Re-test on every model upgrade, and watch for these pitfalls

An embedding benchmark is not a one-off gate you clear before launch — it is a harness you keep and re-run. Re-run it whenever your embedding provider ships a new model version, even one marketed as a drop-in replacement; a same-family upgrade can move recall on your specific queries by several points in either direction, and you will not know which without measuring. Re-run it when your corpus shifts meaningfully — a new document type, a new language added for a UK or India-specific product line, or a large content migration. And re-run it whenever your retrieval architecture changes: a new chunk size, an added reranker, or a rebalanced hybrid-search weighting all interact with the embedding model in ways that are easiest to catch by comparing before-and-after numbers on the same golden set. Our hybrid-retrieval guide covers the reranking and BM25-plus-vector side of that interaction in detail. Absent any of those triggers, a quarterly re-run is a sensible floor.

A handful of pitfalls quietly invalidate a benchmark that otherwise looks rigorous.

A golden set that only covers the easy cases

If every query in your golden set is a clean, well-formed paraphrase of a document's opening sentence, every reasonable model will score well and you will learn nothing about which one handles your harder, messier real traffic — ambiguous phrasing, multi-document questions, queries in a second language. Sample deliberately from your actual query logs once you have them, not just from a tidy synthetic generation pass.

Comparing scores across different chunking strategies

Recall@k, MRR and NDCG are only comparable across models when the underlying chunking, corpus and golden set are held constant. If you change chunk size between two benchmark runs, you are no longer measuring the embedding model — you are measuring the interaction of the model and the chunking change together, and the two effects are impossible to untangle after the fact. Change one variable at a time.

Ignoring cost and latency in the final call

A model with a marginally higher NDCG@10 is not automatically the right choice if it costs four times as much per token or forces a heavier vector index onto your infrastructure. Put the quality numbers next to the cost and latency numbers from the trade-off table above and make the call as a single decision, not two separate ones made by two different people at two different times.

Conclusion: the benchmark is the product, not the model

The teams across India and the UK that get embedding-model selection right are not the ones that read the MTEB leaderboard most carefully. They are the ones that treated the leaderboard as a shortlisting tool, built a modest golden set from their own documents, and kept a small, reusable harness that scores Recall@k, MRR and NDCG on demand. That harness — not the model you pick this quarter — is the durable asset. Models will keep shipping upgrades, licences will keep shifting, and prices will keep moving. A benchmark you can re-run in an afternoon is what lets you catch a regression before your users do, and what lets the next model swap be a two-hour decision instead of a two-week debate.

Start small: a few hundred pairs, three metrics, one afternoon of compute. Grow the harness as you find real failure modes in production. That discipline, more than any single model's MTEB score, is what actually keeps your retrieval quality honest.