A/B testing LLM features: the online experiments that beat offline evals

What you'll set up, in one paragraph

By the end of this guide you will have a repeatable way to answer one question that offline evals cannot: did this change to your LLM feature actually make things better for real people? You will define an Overall Evaluation Criterion — one primary success metric plus a small set of guardrails — assign users to variants with deterministic hashing, protect the experiment with a Sample Ratio Mismatch check, avoid the peeking trap with either a fixed horizon or a sequential test, reach for interleaving when you are changing retrieval, and finally graduate a trusted metric into a contextual bandit that routes traffic automatically. The theme throughout is simple: offline evals prove an output moved; only a live experiment proves it helped.

• Offline tells you it changed; online tells you it helped. Golden sets and LLM-as-judge scores are a gate, not a verdict.
• The OEC is the whole game. One success metric, a few guardrails, agreed before you start.
• Randomise by user, hash deterministically, and check SRM first. A broken split invalidates everything downstream.
• Never peek at a fixed-horizon p-value. Fix the sample size up front, or use always-valid sequential inference.
• Interleave for ranking, A/B for generation, bandit once you trust the OEC.

Why offline evals aren't enough

Offline evaluation has had a very good couple of years, and you should absolutely be doing it. A golden set of representative inputs with graded expected behaviours, an LLM-as-judge to score open-ended answers, and a regression suite wired into continuous integration will stop most obvious disasters before they reach a user. If you have not built that layer yet, start with a proper golden-set and judge harness and wire it into your pipeline so every prompt or model change is regression-tested in CI. That is the cheap, fast, deterministic foundation.

But notice what an offline eval actually measures. It tells you that a new prompt scored 0.82 against your judge where the old one scored 0.79. It does not tell you whether that four-point difference changed a single real decision. Judges have their own biases; golden sets drift away from live traffic the moment you freeze them; and a higher judge score can coincide with longer, waffling answers that users quietly abandon. Picture a Bengaluru fintech that ships a support-bot prompt scoring higher on helpfulness in offline grading — and then watches deflection rate fall in production, because the more "complete" answers bury the one link customers actually needed. The offline number went up. The business number went down.

The right mental model is a flywheel. Modern eval tooling — Braintrust, DeepEval and similar — is built to convert production traces into datasets and test cases, so the outputs of real usage feed back into your offline suite. That flywheel is powerful, and it starts with structured error analysis of your production logs. But even a perfect offline suite is a proxy. When the decision is "do we roll this out to everyone?", the only instrument that answers honestly is a controlled online experiment.

Dimension	Offline eval (golden sets, LLM-judge)	Online experiment (A/B)
Question it answers	Did the output change, and does it look better?	Did it make real users better off?
Ground truth	Fixed labels or a judge model (a proxy)	Actual user behaviour and outcomes
Speed	Minutes, in CI	Days to weeks of live traffic
Cost	A few API calls	Real users exposed to a possibly worse variant
Catches	Regressions, format breaks, obvious wins	Engagement, conversion, retention, trust effects
Blind spot	Judge bias, stale sets, no user reality	Needs enough traffic; slow; novelty effects
Right role	Pre-production gate	The rollout decision itself

Designing the OEC: success plus guardrails

Every good experiment starts with an Overall Evaluation Criterion, or OEC — the single thing you agree in advance to optimise, plus the things you refuse to break. Getting this wrong is the most common reason LLM experiments produce arguments instead of decisions, because without a pre-agreed OEC every stakeholder reads their favourite metric off the dashboard after the fact.

The OEC has two parts. The success metric is the outcome you are trying to move: task-completion rate for an agent, thumbs-up rate for a chat feature, downstream conversion for a shopping assistant, or seven-day retention for a companion product. Choose something as close to real value as your traffic allows. A thumbs-up is easy to measure but weakly correlated with value; a completed booking or a resolved ticket is harder to attribute but far more honest.

The guardrail metrics are the things that must not regress even if the success metric improves: p99 latency, cost per request, refusal rate, and hallucination or safety-flag rate. Guardrails are what stop you shipping a variant that lifts engagement by making the model slower, chattier and three times more expensive. Cost especially deserves its own dashboard — if you cannot see spend per feature and per variant, you are flying blind, so pair every experiment with proper per-feature cost attribution.

Consider a Manchester retailer testing a new product-description generator. The tempting success metric is "click-through on generated descriptions". The honest OEC is: maximise add-to-basket rate (success), provided return rate does not rise, p99 latency stays under 900 ms, and cost per description stays under the current model's (guardrails). That sentence, agreed by product, engineering and finance before the test starts, is worth more than any dashboard you build later.

Assignment, randomisation and SRM

Once the OEC is fixed, the mechanics of assignment decide whether your result means anything. Three choices matter: the randomisation unit, how you compute assignment, and how you verify the split.

Randomisation unit. For anything conversational, randomise by user, not by request. If you assign per request, a single London insurance customer can bounce between the control prompt and the treatment prompt inside one conversation — the treatment leaks, the conversation-level metric is meaningless, and the experience is jarring. User-level assignment keeps a person in one arm across every turn, session and device. Session-level is an acceptable fallback only when you have no durable user id, and request-level is right only for genuinely stateless, single-shot calls.

Deterministic hashing. Do not flip a coin at request time and store it — that invites drift and race conditions. Hash a stable identifier together with the experiment name, and map the hash into a bucket. The same user always lands in the same arm, assignment is reproducible for debugging, and you can add experiments without disturbing existing ones.

import hashlib

def assign(user_id: str, experiment: str, weights=(("control", 50), ("treatment", 50))) -> str:
    # Deterministic 0..9999 bucket from a stable id + experiment salt.
    key = f"{experiment}:{user_id}".encode("utf-8")
    bucket = int(hashlib.sha256(key).hexdigest(), 16) % 10_000

    cumulative, total = 0, sum(w for _, w in weights)
    for arm, w in weights:
        cumulative += w * 10_000 // total
        if bucket < cumulative:
            return arm
    return weights[-1][0]

# Same user -> same arm, every request, across sessions and devices.
assert assign("user_8842", "prompt_v7") == assign("user_8842", "prompt_v7")

The SRM check. Before you read a single success metric, verify that the observed traffic split matches the split you configured. A Sample Ratio Mismatch — say 51.8/48.2 when you asked for 50/50 — is a chi-squared test away from being detected, and when it fires it almost always means a bug: a redirect dropping traffic on one arm, logging that only fires for treatment, bots concentrated in one bucket, or an assignment that runs after an early return. Treat SRM as a trust gate. If it fails, the experiment is void until you find the cause, no matter how beautiful the lift looks.

from scipy.stats import chisquare

def srm_check(n_control: int, n_treatment: int, expected_ratio=0.5) -> dict:
    total = n_control + n_treatment
    expected = [total * expected_ratio, total * (1 - expected_ratio)]
    stat, p = chisquare([n_control, n_treatment], f_exp=expected)
    return {
        "p_value": p,
        "srm_detected": p < 0.001,   # strict: a broken split is a hard stop
        "observed_split": (n_control / total, n_treatment / total),
    }

# srm_detected == True  ->  DO NOT read the metrics. Find the bug first.

The peeking problem and sequential tests

Here is the trap that quietly ruins more LLM experiments than any other. You set up a clean A/B test, you watch the dashboard, and on day three the p-value dips below 0.05. You call it, ship the winner, and move on. The problem is that a fixed-horizon t-test assumes you look exactly once, at a pre-committed sample size. Every extra peek is another roll of the dice, and if you stop at the first significant reading your true false-positive rate is not 5% — it can easily be 20% or worse. You will ship variants that did nothing, celebrate, and wonder later why the aggregate metric never moved.

There are two disciplined ways out.

Fix the horizon. Before launching, do a power calculation: decide the minimum detectable effect (MDE) you care about, pick your power (usually 80%) and significance level (usually 5%), and compute the sample size that combination requires. Then run to that sample size and read the result once. No peeking. This is the simplest correct approach, and it forces a healthy conversation about how small an effect is even worth chasing.

There is an LLM-specific wrinkle here. Because generation is non-deterministic — temperature, sampling, and the sheer variety of open-ended outputs — your success metric carries more variance than a typical button-colour test. Higher variance means a larger sample size for the same MDE. Budget for it. If you want a tighter test, reducing temperature on the measured path or scoring with a more stable rubric can shrink the variance you have to overcome.

Use sequential testing. If you genuinely need to monitor continuously and stop as soon as you have an answer, use a method designed for it — a mixture Sequential Probability Ratio Test (mSPRT) or another always-valid inference approach that produces confidence sequences safe to check at any time. These trade a little power for the freedom to peek honestly, which is often the right trade for a fast-moving product team.

# mSPRT-style always-valid test (conceptual pseudocode).
# Unlike a fixed-horizon t-test, this may be checked after every batch.

state = SequentialTest(alpha=0.05, mde=0.02, variance_prior=sigma_sq)

for batch in stream_of_conversions():          # arrives continuously
    state.update(batch.control, batch.treatment)

    if state.always_valid_p < state.alpha:      # safe to peek — no inflation
        decide("treatment wins", lift=state.effect)
        break
    if state.futility_boundary_crossed():        # unlikely to ever win
        decide("no effect — stop", lift=state.effect)
        break
    # else: keep collecting; the boundary already accounts for every look

Whichever route you pick, decide it before launch and write it into the OEC doc. The cardinal sin is running a fixed-horizon test and then peeking anyway — you get the worst of both worlds.

Interleaving versus A/B

Not every change needs a full between-users A/B test, and for one important class of change there is a sharper instrument. If you are altering retrieval or ranking — a new embedding model, a reranker, a different chunking strategy in your RAG pipeline — you can use interleaving. Instead of showing user A the old ranker and user B the new one, you blend both rankers' results into a single list for the same user and observe which side's items earn the clicks. Because every user is effectively their own control, interleaving removes between-user variance and reaches a verdict with dramatically fewer users and less time. A Delhi marketplace can settle a reranker question in a day with interleaving that would take a fortnight as a classic A/B test.

Interleaving does not work for generative changes. You cannot blend two whole answers, two prompt styles, or two model personalities into one coherent response, so a new system prompt, a new answer format or a new model is a job for a classic between-users A/B test on your OEC. The rule of thumb: interleave when the output is a ranked list; A/B when the output is generated prose or a UX change.

From experiment to bandit routing

An A/B test is the right tool when you need a trustworthy, defensible answer to a single question. But once you genuinely trust your OEC — once you have watched it move in tests and confirmed it tracks real value — you can let the OEC drive traffic automatically. That is what a contextual bandit does: rather than a fixed 50/50 split for a fixed horizon, it continuously shifts traffic toward whichever variant is performing best for a given context, balancing exploration and exploitation.

For LLM features this is powerful, because you frequently have several viable variants — a fast cheap model, a slow accurate one, two prompt styles — and the best choice depends on context: query type, user tier, language, time of day. A contextual bandit can route a straightforward FAQ to the cheap model and a complex, high-value query to the expensive one, learning the boundary from the OEC itself. A Pune SaaS support team might let a bandit choose between three answer templates per ticket category, while a UK bank keeps everything on a fixed A/B test for auditability.

Two cautions. First, a bandit is only as good as its reward signal — if the OEC is noisy or gamed, the bandit optimises the wrong thing faster than any human could. Earn trust with A/B tests before you hand the wheel over. Second, bandits make clean causal read-outs harder, because assignment is no longer independent of performance; keep a small holdout on a fixed split if you still need a defensible lift number for stakeholders or regulators.

Putting it together

The workflow that holds up in production is a ladder, not a single rung. Offline evals and CI regression tests are the cheap, fast pre-production gate. A shadow or canary deploy is the safety check that protects users from latency, cost and correctness regressions. The A/B test — with a pre-agreed OEC, user-level deterministic assignment, an SRM trust gate, and a peeking-safe stopping rule — is the instrument that actually decides whether you roll out. Interleaving sharpens that decision for ranking changes. And once the OEC has earned your trust, a contextual bandit turns the whole thing into an automatic router.

Do that, and you stop shipping AI features because a judge score ticked up. You ship them because you watched real users in India and the UK become measurably better off — and you can prove it. That is the difference between an output that changed and a product that improved.