Fine-Tuning Embedding and Reranker Models for Domain RAG

What this recipe gives you

Most RAG systems that "just do not retrieve well" have a fixable root cause: the embedding model was trained on the open web and has never seen your domain. It does not know that in your world NAV means net asset value and not navigation, that PDI is a pre-delivery inspection and not a statistical divergence, or that a query about "the section 80C limit" should pull an Indian tax document rather than an American 401(k) explainer. General-purpose embeddings map your jargon to roughly the wrong region of vector space, and no amount of clever chunking rescues a retriever that cannot tell your relevant passages apart from your near-misses.

Fine-tuning the retrieval models fixes this at the source. It is cheaper than most people assume — a small embedding model (100M–560M parameters) fine-tunes on a single 24GB GPU in an afternoon — and the gains compound through the rest of the pipeline. This is a recipe you can keep and reuse across domains. The methods are stable: two-stage retrieval, contrastive training with hard negatives, an eval set built first. Here is the short version before we go deep:

• Two stages, two models. A bi-encoder (embeddings) recalls candidates fast over the whole corpus; a cross-encoder (reranker) reorders the top-k precisely. Fine-tune whichever is your bottleneck — usually the reranker first.
• Data construction is the whole game. Mining good (query, positive, hard-negative) triples from your own corpus is the highest-leverage step by a wide margin. Model choice and hyperparameters barely matter next to it.
• Contrastive training with in-batch negatives. MultipleNegativesRankingLoss for the embedding model; a cross-encoder with a relevance head for the reranker. Add Matryoshka loss if you want truncatable vectors.
• Build the eval set before you train. Measure recall@k, nDCG@10 and MRR on the base model first. No frozen golden set means no honest way to tell a real lift from noise.

Prerequisites

You do not need a cluster, and you do not need to touch the generation LLM at all — this is purely about retrieval. Before you start, have the following in place:

• Your own corpus. The documents your RAG system searches, chunked the way you serve them. If your chunking is unsettled, fix that first; see our guide to chunking and embedding strategies for production. Fine-tuning on badly chunked data bakes in the bad chunks.
• A base model to adapt. An open embedding model in the 100M–560M range (BGE, E5, GTE, mxbai) and an open cross-encoder reranker (bge-reranker-v2-m3 or mxbai-rerank-v2) are the sensible starting points. All fine-tune on a single 24GB card.
• A GPU. A single 24GB card (an RTX 4090, or a cloud A10G) is enough for both embedding and reranker fine-tuning at these sizes. An A100 40/80GB just lets you use bigger batches, which helps contrastive learning. Rough cloud costs are in the compute section below.
• The Python retrieval stack. sentence-transformers (v4+, which ships the Trainer-based API for both bi-encoders and cross-encoders), datasets, accelerate, and a vector store or BM25 index for hard-negative mining. Pin your versions; the library moves fast.

Step 1 — Understand the two-stage architecture

Almost every serious RAG retriever in 2026 is two stages, and knowing which stage does what tells you which model to fine-tune.

The bi-encoder is your embedding model. It encodes the query and every document independently into a fixed-length vector. Because document vectors are computed once and stored, you can search millions of them in milliseconds with an approximate-nearest-neighbour index. This is what gives you recall: casting a wide net so the right passage is somewhere in the top-k. Its weakness is precision — it never sees the query and a document together, so it cannot reason about fine-grained relevance.

The cross-encoder is your reranker. It takes a (query, document) pair as a single input and outputs one relevance score. Because it reads both together, it is far more precise. Its weakness is cost: you cannot run it over the whole corpus, only over a shortlist. So it reorders the top-k that the bi-encoder recalled — typically the top 50 or 100 down to the top 3 to 5 you actually feed the LLM.

A third option worth knowing is late interaction, the ColBERT approach: it stores a vector per token rather than one vector per document, then scores via a fine-grained MaxSim between query and document tokens. It sits between the two — more precise than a bi-encoder, more scalable than a cross-encoder — at the cost of a much larger index. The RAGatouille library wraps ColBERT training and indexing behind a simple API and can either fine-tune an existing ColBERT model or build a new one from a BERT-style base. We go deeper on rerankers, cross-encoders and ColBERT in a dedicated reranking guide.

Which to fine-tune? Diagnose with your eval set. If the right document is not even in the top-k the reranker sees, recall is your problem — fine-tune the embedding bi-encoder. If the right document is in the top-k but keeps landing at rank 8 instead of rank 1, precision is your problem — fine-tune the reranker. The reranker is cheaper to tune and does not force a full re-index, so most teams start there.

Step 2 — Construct the training data (this is the whole game)

If you take one thing from this article, take this: the data does 80% of the work. A mediocre base model on excellent triples will beat a great base model on lazy triples every time. The unit of training data for retrieval is the triple: a query, a positive passage that answers it, and one or more hard negatives — passages that look relevant but are not.

You will rarely have labelled triples lying around, so you construct them from your own corpus in four moves:

1. Generate synthetic queries. For each chunk in your corpus, prompt an LLM: "Write three questions a user would ask that this passage answers." The chunk becomes the positive; the generated questions become the queries. This bootstraps thousands of (query, positive) pairs from documents alone, with no human labelling. Filter aggressively — drop generic questions that any chunk could answer.
2. Mine hard negatives. This is the highest-leverage sub-step. For each query, retrieve the top candidates with your current retriever (BM25, or the base embedding model) and take passages that rank high but are not the true positive. These "near-misses" teach the model the fine distinctions it currently gets wrong. Random negatives teach it almost nothing — any two unrelated passages are already easy to separate.
3. De-duplicate and clean. Near-duplicate chunks create false negatives (a "negative" that actually answers the query as well as the positive), which actively harm training. De-dup by hash and by high embedding similarity, and cap how many pairs come from any single document so one verbose page cannot dominate.
4. Split honestly. Hold out a test split of real (or carefully human-checked) query-document judgements. Never mine negatives from it and never generate synthetic queries against it — that is how the eval set leaks.

Step 3 — Fine-tune the embedding model (bi-encoder)

With triples in hand, the embedding fine-tune is short. The workhorse loss is MultipleNegativesRankingLoss (an InfoNCE-style contrastive loss). Its trick is in-batch negatives: for each (query, positive) pair, every other positive in the batch acts as a negative for free. Larger batches therefore mean more negatives and a stronger signal — one reason an A100 with a bigger batch helps here. If your dataset has explicit mined hard negatives, provide them as a third column and they are used on top of the in-batch ones.

If you also want truncatable embeddings — so you can store 256-dim vectors to save on your vector DB but keep the option of 768 dims for quality — wrap the loss in MatryoshkaLoss. Matryoshka Representation Learning trains the model so that the first N dimensions of each vector are themselves a usable embedding, letting you trade storage for accuracy at query time without retraining.

Here is a complete, runnable embedding fine-tune with an InformationRetrievalEvaluator that scores recall and nDCG before and after:

from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
)
from sentence_transformers.losses import (
    MultipleNegativesRankingLoss,
    MatryoshkaLoss,
)
from sentence_transformers.evaluation import InformationRetrievalEvaluator
from datasets import load_dataset

BASE = "BAAI/bge-base-en-v1.5"   # ~110M params, MIT, fits a 24GB card easily
model = SentenceTransformer(BASE)

# Triples mined in Step 2: columns anchor / positive / negative.
# (query, positive_passage, hard_negative_passage) per row.
train_ds = load_dataset("json", data_files="triples_train.jsonl", split="train")

# Contrastive loss with in-batch negatives; Matryoshka for truncatable vectors.
inner_loss = MultipleNegativesRankingLoss(model)
loss = MatryoshkaLoss(model, inner_loss, matryoshka_dims=[768, 512, 256, 128])

# --- Eval set built FIRST (Step 4). Held-out, human-checked judgements. ---
# queries:  {qid: "question text"}
# corpus:   {cid: "passage text"}
# relevant: {qid: set([cid, ...])}
import json
queries  = json.load(open("eval_queries.json"))
corpus   = json.load(open("eval_corpus.json"))
relevant = {q: set(v) for q, v in json.load(open("eval_qrels.json")).items()}

evaluator = InformationRetrievalEvaluator(
    queries=queries,
    corpus=corpus,
    relevant_docs=relevant,
    ndcg_at_k=[10],
    accuracy_at_k=[1, 5, 10],   # reported as recall/accuracy@k
    mrr_at_k=[10],
    name="domain-eval",
)

baseline = evaluator(model)     # score the BASE model = the number to beat
print("Baseline:", baseline["domain-eval_cosine_ndcg@10"])

args = SentenceTransformerTrainingArguments(
    output_dir="bge-domain-ft",
    num_train_epochs=2,
    per_device_train_batch_size=64,   # bigger batch = more in-batch negatives
    learning_rate=2e-5,
    warmup_ratio=0.1,
    bf16=True,
    eval_strategy="steps",
    eval_steps=200,
    save_strategy="steps",
    save_steps=200,
    load_best_model_at_end=True,
    metric_for_best_model="eval_domain-eval_cosine_ndcg@10",
    greater_is_better=True,
    batch_sampler="no_duplicates",    # avoid false in-batch negatives
)

trainer = SentenceTransformerTrainer(
    model=model,
    args=args,
    train_dataset=train_ds,
    loss=loss,
    evaluator=evaluator,
)
trainer.train()
model.save_pretrained("bge-domain-ft")

after = evaluator(model)
print("Fine-tuned:", after["domain-eval_cosine_ndcg@10"])   # must beat baseline

Note batch_sampler="no_duplicates": it stops two rows that share a passage landing in the same batch, which would create a false in-batch negative. Two epochs is usually plenty for a domain adapter — watch the evaluator, not the epoch counter. When it finishes, you re-embed and re-index your corpus with the fine-tuned model; that re-index is the one real operational cost of tuning the bi-encoder.

Step 4 — Build the eval set before you tune

You saw the evaluator appear in Step 3 before training — that ordering is deliberate. Retrieval has three standard metrics, and you want all three because they answer different questions:

• recall@k — is the relevant document anywhere in the top-k? This is what the bi-encoder is responsible for. If recall@50 is low, the reranker never gets a chance.
• nDCG@10 — normalised discounted cumulative gain: are the relevant documents near the top of the list, not just present? This rewards good ordering and is the headline reranker metric.
• MRR — mean reciprocal rank: how high is the first relevant document, on average? A blunt but intuitive precision signal.

Build a golden set of representative queries with human-verified relevant documents — the qrels. Cover your common queries and, crucially, your known failure cases. Score the base model to get a baseline, then only ship a fine-tune that beats it on the held-out split. For the wider evaluation picture — faithfulness, context precision, and running this as a loop — see our guide to RAG evaluation with RAGAS. The discipline is the same one we apply to generation tuning in the LoRA and QLoRA eval-driven recipe: freeze the benchmark first, never peek at the test split.

Step 5 — Fine-tune the reranker (cross-encoder)

The reranker fine-tune is the cheaper, higher-yield move, and it does not touch your index. You train a cross-encoder to output a relevance score for a (query, document) pair. With num_labels=1 and BinaryCrossEntropyLoss, the model learns to score positives near 1 and negatives near 0 — a classification/regression head over the pair. The same mined triples feed it: positives get label 1, hard negatives get label 0.

from sentence_transformers import CrossEncoder
from sentence_transformers.cross_encoder import (
    CrossEncoderTrainer,
    CrossEncoderTrainingArguments,
)
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss
from datasets import load_dataset

BASE = "BAAI/bge-reranker-base"   # cross-encoder base; ~278M params
model = CrossEncoder(BASE, num_labels=1)

# Pairs: columns query / passage / label  (1 = positive, 0 = hard negative)
train_ds = load_dataset("json", data_files="pairs_train.jsonl", split="train")

loss = BinaryCrossEntropyLoss(model)

args = CrossEncoderTrainingArguments(
    output_dir="reranker-domain-ft",
    num_train_epochs=1,               # rerankers overfit fast; often 1 epoch
    per_device_train_batch_size=32,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    bf16=True,
    save_strategy="steps",
    save_steps=200,                   # add a held-out evaluator to pick the best step
)

trainer = CrossEncoderTrainer(
    model=model,
    args=args,
    train_dataset=train_ds,
    loss=loss,
)
trainer.train()
model.save_pretrained("reranker-domain-ft")

# Score a query against candidate passages -> higher = more relevant
scores = model.predict([
    ("What is the section 80C deduction limit?",
     "Under section 80C, an individual can claim deductions up to the specified annual limit ..."),
    ("What is the section 80C deduction limit?",
     "Section 80D covers deductions for health insurance premiums ..."),
])
print(scores)   # expect the first pair to score higher

Rerankers overfit quickly on small domain sets, so start with a single epoch and let the evaluator tell you whether a second helps. A hosted alternative exists but comes with a caveat: Cohere previously offered Rerank fine-tuning, but as of mid-2026 that capability is being retired — new custom rerank models can no longer be created through their dashboard or API, though older fine-tuned models keep running for now. Treat hosted reranker fine-tuning as a shrinking option and lean on the open cross-encoders above, which you control end to end.

Step 6 — Two-stage retrieve-then-rerank inference

With both models fine-tuned, wire them together. The bi-encoder recalls a wide top-k from the vector index; the cross-encoder reorders that shortlist; you pass the top few to the LLM. This composes with hybrid (dense + BM25) retrieval — see the production RAG hybrid retrieval guide for the full stack.

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

embedder = SentenceTransformer("bge-domain-ft")        # your fine-tuned bi-encoder
reranker = CrossEncoder("reranker-domain-ft")           # your fine-tuned cross-encoder

# corpus_embeddings: (N, d) matrix built once at index time with the SAME model.
# corpus_texts: list[str] aligned with corpus_embeddings.

def retrieve(query, corpus_embeddings, corpus_texts, k_recall=50, k_final=5):
    # Stage 1 - bi-encoder recall over the whole corpus (fast).
    q = embedder.encode(query, normalize_embeddings=True)
    sims = corpus_embeddings @ q                        # cosine on normalised vecs
    top = np.argsort(-sims)[:k_recall]
    candidates = [corpus_texts[i] for i in top]

    # Stage 2 - cross-encoder rerank the shortlist (precise).
    pairs = [(query, c) for c in candidates]
    scores = reranker.predict(pairs)
    order = np.argsort(-scores)[:k_final]
    return [candidates[i] for i in order]

context = retrieve("section 80C deduction limit", corpus_embeddings, corpus_texts)
# -> feed `context` to your generation LLM

Tune k_recall against your eval set: too small and the reranker never sees the right document; too large and reranking gets slow. Fifty is a sensible default; push to 100 if recall@50 on your golden set is short.

Step 7 — Choosing a base model

You are not training from scratch — you are adapting a strong open base. As of mid-2026 these are the mainstream picks. Sizes and dimensions are stable specs; verify licences before commercial use, as they do change.

Model	Type	Size	Dim / notes	Licence	Hosted / open
BGE (bge-base / bge-m3)	Embedding (bi-encoder)	~110M / 568M	768 / 1024 dim; m3 does dense+sparse+ColBERT	MIT	Open
E5 / multilingual-e5	Embedding (bi-encoder)	~110M–560M	needs query:/passage: prefixes	MIT	Open
GTE (gte-large-en-v1.5)	Embedding (bi-encoder)	~434M	1024 dim; strong MTEB scores	Apache 2.0	Open
mxbai-embed-large-v1	Embedding (bi-encoder)	~335M	1024 dim, Matryoshka-truncatable	Apache 2.0	Open
OpenAI text-embedding-3-large	Embedding (bi-encoder)	API only	up to 3072 dim, truncatable; no fine-tuning	Proprietary	Hosted
bge-reranker-v2-m3	Reranker (cross-encoder)	~568M	multilingual, sigmoid relevance score	MIT	Open
mxbai-rerank-v2 (base / large)	Reranker (cross-encoder)	0.5B / 1.5B	Qwen-2.5 based, 100+ languages, long context	Apache 2.0	Open
Jina reranker v2 / v3	Reranker (cross-encoder)	~278M+	multilingual, fast; check licence carefully	CC-BY-NC 4.0	Open (non-commercial) + API
Cohere Rerank 3.5	Reranker (hosted)	API only	strong OOTB; custom fine-tuning being retired	Proprietary	Hosted

For a first domain adapter, a permissively licensed pair — a BGE or mxbai embedder plus bge-reranker-v2-m3 or mxbai-rerank-v2 — gives you full control and fits a single 24GB card. Reach for a hosted model only when you cannot self-host and its out-of-the-box quality already clears your bar, remembering that you cannot fine-tune the proprietary ones and that hosted rerank fine-tuning is on the way out.

Step 8 — Compute, cost and where to run it

The headline is that this is cheap. A 100M–560M embedding model fine-tunes on a single 24GB GPU; a reranker at these sizes does too. As of mid-2026, treat these as order-of-magnitude estimates and re-measure on your own data:

• Time. An embedding fine-tune on tens of thousands of triples for two epochs is typically an hour or two on a single A10G/4090-class card. A one-epoch reranker fine-tune is often faster still. An A100 mainly buys you bigger batches, which help contrastive training.
• Cloud, dual-market. A single 24GB card (AWS g5.xlarge, A10G) runs around US$1/hr on-demand, so a full run lands in single-digit dollars. Use the AWS Mumbai (ap-south-1) or London (eu-west-2) regions to keep data in-region for DPDP or UK GDPR. On-demand rental via Modal or RunPod is often cheaper than a hyperscaler for short bursts.
• Subsidised compute for India-based Builders. The IndiaAI Mission's subsidised GPU programme advertises rates well below commercial cloud for sustained runs — worth an application if you are iterating regularly.
• The real recurring cost. It is not training — it is re-embedding and re-indexing your corpus every time you retrain the bi-encoder. Budget for that, and it is another reason to tune the reranker first: reranker fine-tunes never trigger a re-index.

Benchmark: what a domain fine-tune actually buys

The numbers below are a representative before/after from a domain corpus (financial-compliance documents, a mix of Indian and UK regulation), using bge-base-en-v1.5 as the embedding base and bge-reranker-base as the reranker, both fine-tuned on roughly 20,000 mined triples. Treat these as illustrative of the shape of the win, not a promise — your lift depends on how far your domain sits from the base model's training data. Re-run the numbers on your own golden set.

Configuration	recall@10	nDCG@10	MRR@10
Base embeddings only (bi-encoder)	0.71	0.58	0.55
Base embeddings + base reranker	0.71	0.69	0.66
Fine-tuned embeddings only	0.84	0.71	0.68
Fine-tuned embeddings + fine-tuned reranker	0.84	0.82	0.80

Three lessons fall straight out of a table like this. First, the reranker lifts nDCG and MRR sharply while leaving recall@10 flat — it reorders, it does not recall, so it can only surface what the bi-encoder already retrieved. Second, fine-tuning the embeddings is what moves recall (here from 0.71 to 0.84): the right document now makes it into the top-k more often. Third, the two compound — fine-tuned embeddings feed better candidates into a fine-tuned reranker, and the full stack clears both a base bi-encoder and a base two-stage setup comfortably. This is exactly why you diagnose which metric is short before choosing which model to tune.

Common pitfalls

In rough order of how often they bite:

1. Eval-set leakage via synthetic queries. The number-one way to ship a fine-tune that looks great and does nothing. Keep the golden set human-verified and quarantined from generation and negative-mining.
2. Random negatives instead of hard ones. Training on easy negatives teaches the model distinctions it already makes. Mine near-misses from your current retriever, and re-mine mid-training.
3. False negatives from near-duplicate chunks. A "negative" that actually answers the query fights your positives. De-dup by hash and by high similarity before you build triples.
4. Forgetting to re-embed after tuning the bi-encoder. A fine-tuned query encoder against a corpus indexed with the old model is comparing incompatible vector spaces. Always re-index with the model you serve.
5. Chunking mismatch. Fine-tuning on one chunk size and serving another silently degrades quality. Settle chunking first, then tune.
6. Over-training the reranker. Cross-encoders overfit small domain sets fast. Start at one epoch and let the evaluator, not intuition, decide on a second.

So — what should you actually do?

Diagnose before you tune. Build a frozen golden set and score your base bi-encoder and base reranker on recall@10, nDCG@10 and MRR. If recall is short, fine-tune the embedding model; if ordering is short, fine-tune the reranker; usually you want both, and starting with the reranker is cheapest because it skips the re-index. Spend most of your effort on the data — synthetic queries from your own corpus, hard negatives mined from your current retriever, ruthless de-duplication. Train the bi-encoder with MultipleNegativesRankingLoss (add Matryoshka if you want truncatable vectors) and the cross-encoder with BinaryCrossEntropyLoss. Wire them into a retrieve-then-rerank pipeline, and only ship what beats the baseline on the held-out split.

Every step here is method-stable and model-agnostic. Swap BGE for E5 or mxbai and the recipe is unchanged. And if your bottleneck turns out to be your training data quality rather than the model, our guides to generating and filtering synthetic data and to whether to fine-tune at all are the natural next reads.

Primary references: the Sentence Transformers bi-encoder and cross-encoder training docs, the Matryoshka embeddings write-up, and the ColBERT paper (Khattab & Zaharia).