Re-Qualify a Model After a Price Cut: a Three-Day Drill

Why a price cut is a trap as well as an opportunity

A forced migration has one redeeming feature: a deadline. When a model is deprecated, when a contract ends, when a region loses availability, somebody owns the date and the work gets done. A price cut has no such feature. Nothing is broken. Yesterday's system still works today at yesterday's quality, and the only thing that changed is that a cheaper option now exists. That is precisely why so many of these migrations never happen. They sit in the backlog as "look at the new pricing", get carried across three sprints, and are quietly closed in the next quarterly clean-up with the saving unrealised.

The mirror-image failure is worse and less visible. Somebody with spare capacity swaps the model identifier in a configuration file on a Friday, the dashboards do not move, and a subtle quality regression seeps into production over the following month. Nobody rolls it back because nobody connected the two events. The cheap win became a slow, unattributed degradation, which is the single most expensive outcome available.

So the discipline here is not thoroughness. It is a box. Three working days, a frozen task set, one candidate model, one scaffold, and a written rule set agreed before anybody sees a result. At the end of day three you either move a defined slice of traffic, or you write one dated paragraph saying why not. Both are acceptable. Drifting is not.

The second reason to be careful is arithmetic. The headline percentage is almost never the number you save. As of September 2026, Anthropic listed Claude Opus 5.5 at four US dollars per million input tokens and twenty dollars per million output tokens, down from Opus 5's five and twenty-five — a twenty per cent cut on list — while claiming an effective cost roughly forty per cent lower on the grounds that the newer model uses fewer tokens per task and produces output faster. Over the same period OpenAI moved GPT-6 Sol to two dollars per million input and ten dollars per million output, against GPT-5.6 Sol's four and twenty, and GPT-6 Luna to ten cents per million input and fifty cents per million output, against Luna's earlier twenty cents and one dollar twenty, confirming these as permanent rather than promotional prices.

Read those two announcements carefully and you will notice they are making different kinds of claim. One is a list-price change, which is arithmetic you can verify from an invoice. The other is an effective-cost claim, which is a statement about a workload mix that may or may not resemble yours. A halving of list price on a workload whose bill is eighty per cent cached input behaves very differently from the same cut on an agentic workload that burns output tokens across a dozen turns. Neither vendor is being misleading; the effective number is simply not a property of the price list, it is a property of your traffic. The drill exists to measure that property.

Before day one: what has to already exist

The three-day box only works if four things are already in place. None of them are exotic, but if you are missing them, be honest about it: the drill becomes a week, and the first job is building the missing piece rather than pretending it away.

A frozen, representative task set. Between two hundred and four hundred items drawn from real production traffic, each with a programmatic check that returns a pass or a graded score without human judgement in the loop. Frozen means nobody edits it during the drill, because a task set that moves while you are measuring makes every comparison meaningless. If you do not have one, build it first — the mechanics of golden sets, graders and judges are covered in our guide to building your first LLM evaluation suite, and that build is a week of work, not an afternoon.

Logged production traces with token counts. Not just latencies and error rates: input tokens, output tokens, cached-read tokens, cache-write tokens, tool-call counts and turn counts, per request, retained for at least a month. Without these you cannot construct a cost-per-task baseline and you will end up estimating from the invoice, which averages across everything and hides exactly the variation you need to see.

A cost-per-task baseline. The current model's cost to complete one unit of useful work, by task family, computed from those traces. Not cost per token, not cost per request — cost per completed task, including retries and tool turns.

A rollback switch that is configuration, not deployment. You must be able to send traffic back to the incumbent in under a minute without shipping code. If reverting requires a release, you do not have a rollback switch, you have a rollback project, and you will hesitate to cut over at all.

Day 1: establish the honest baseline

Day one touches only the incumbent. You are not running the new model yet; you are working out what you actually pay today, which is a surprisingly common gap even on teams with good dashboards.

Pull a fortnight of production traces and compute, per task family: median and p95 input tokens, median and p95 output tokens, the cached-read share of input, the number of model calls per completed task, the retry rate, the success rate against your programmatic check, and the wall-clock time from first request to completed task. Then price all of it at the rate you actually pay, in the currency your budget is denominated in.

The reason to measure tokens per task and wall-clock rather than price per token is that the two can point in opposite directions. A model can be cheaper per token and more expensive per task. It happens through three mechanisms. It may be more verbose, producing longer answers for the same work, and output tokens are typically the expensive side of the price list. It may take more agent turns, because a weaker planner re-reads context and re-issues tool calls, and every turn re-sends the prompt. Or it may be less reliable on your check, so the retry rate rises and you pay twice for a share of your traffic. Any of the three can eat a twenty per cent list-price cut without appearing anywhere on a cost dashboard.

Wall-clock matters for the same reason, plus one more. If your workload is agentic and you pay for a container, a worker or a human's attention while the model thinks, slower output is a real cost that never appears on the token invoice. A vendor claiming faster output as part of an effective-cost improvement is making a claim you can verify directly, and should.

Note what the illustrative table does to the headline. A twenty per cent list cut became a thirty-four per cent reduction per attempt because the candidate was assumed to use fewer turns and fewer tokens, then fell back to thirty per cent per successful task because the assumed quality regression bought back some retries. Every one of those movements is invisible if you compare price lists, and every one of them is measurable in two days. Change the invented success rates to ninety-two and eighty, and the same table flips to a far less attractive saving; change the turn count to identical, and the saving collapses towards the list-price figure. That sensitivity is the whole argument for running the drill.

One dual-market note before you leave day one. Price lists are almost always denominated in US dollars while your budget is not. An Indian team reporting in rupees and a UK team reporting in pounds are both exposed to a currency move that can be comparable in size to the price cut itself over a year. Record the exchange rate and the date alongside every figure, and if you are presenting a saving to a finance lead, present it in both the invoice currency and the reporting currency. Endpoint geography deserves the same treatment: the same model served from an AWS Mumbai region and a London region will not give you the same p95, and if the cheaper model is not yet available in the region you are required to use, that is a finding that ends the drill on day one.

Day 2: run the candidate side by side

Day two is a paired comparison and nothing else. Same inputs, same scaffold, same tools, same retry policy, same temperature discipline, same grader. The only thing that changes is the model identifier.

What to hold constant

Hold constant: the system prompt, the tool definitions and their descriptions, the retrieval configuration and the number of documents retrieved, the maximum token limits, the stop conditions, the retry budget, the grading code, and the task set itself. Hold constant even the things you are confident do not matter, because the value of a paired design comes entirely from there being exactly one difference between the two arms.

Vary deliberately and record: nothing, on day two. If you find yourself wanting to change the scaffold to make the candidate look better, write the change down as a day-four hypothesis and carry on.

The trap of prompts tuned for the old model

This is the sharpest edge in the whole drill, and it cuts both ways. Your production prompt has been shaped over months against the incumbent. It may contain instructions that exist only to suppress a habit the incumbent had — "do not apologise", "do not restate the question", "always close the JSON object" — and those instructions are now either inert or actively harmful. It may contain few-shot examples in the incumbent's preferred output style. It may rely on a system-prompt behaviour the candidate handles differently.

The honest framing is that a paired run with the incumbent's prompt measures drop-in replaceability, which is exactly the question a price-cut migration asks. It does not measure the candidate's ceiling. Report it as such. If the candidate fails on the incumbent's prompt but you suspect the prompt is the cause, the correct output of the drill is a shortlist verdict with a named follow-up, not a quiet decision to tune the prompt mid-experiment and re-run.

Why distributions, not single runs

Two aggregate scores tell you almost nothing, because the variance between tasks in a realistic set is usually far larger than the difference between two competent models. Pairing removes that variance: for each item, you have both models' results on identical input, so you can look at the distribution of per-item differences instead. Most items will be ties. The signal lives in the discordant pairs, where one model passed and the other failed.

For a binary check, count the discordant pairs in each direction and run an exact binomial test on them. Below roughly twenty-five disagreements you generally cannot separate a real regression from noise, and the correct report is that the drill was underpowered — not that the models are equivalent. For graded scores, bootstrap the mean paired difference and report an interval rather than a point. Run three repeats per item per model so that you can also see within-model variance, which is your noise floor for everything else.

"""Day 2: paired A/B evaluation over a frozen golden set.

Provider-agnostic on purpose: you supply two `run_fn` callables, one per arm.
Each must have signature run_fn(item) -> (output_text, usage_dict) where
usage_dict has integer keys: input_tokens, output_tokens, cached_input_tokens,
calls. Wire them to whatever SDK you use -- the harness never sees it.

The output is a paired difference distribution plus an exact binomial test on
the discordant pairs, NOT two aggregate scores.
"""
from __future__ import annotations

import math
import random
import statistics
import time
from dataclasses import dataclass, field


@dataclass
class ItemResult:
    item_id: str
    passed: bool
    score: float
    latency_ms: float
    usage: dict


@dataclass
class Arm:
    name: str
    results: list = field(default_factory=list)

    def by_item(self) -> dict:
        return {r.item_id: r for r in self.results}


def run_arm(name, items, run_fn, grade_fn, repeats=3, max_attempts=2) -> Arm:
    """Identical control flow for both arms. Retries are inside the arm,
    because retries are inside your bill."""
    arm = Arm(name=name)
    for item in items:
        for _ in range(repeats):
            started = time.perf_counter()
            passed, score = False, 0.0
            usage = {"input_tokens": 0, "output_tokens": 0,
                     "cached_input_tokens": 0, "calls": 0}
            for _attempt in range(max_attempts):
                output, u = run_fn(item)
                for k in usage:
                    usage[k] += int(u.get(k, 0))
                passed, score = grade_fn(item, output)
                if passed:
                    break
            arm.results.append(ItemResult(
                item_id=item["id"],
                passed=passed,
                score=float(score),
                latency_ms=(time.perf_counter() - started) * 1000.0,
                usage=usage,
            ))
    return arm


def discordant_pairs(baseline: Arm, candidate: Arm) -> tuple:
    """Collapse repeats to a majority verdict per item, then count the items
    where the two arms disagree. Ties carry no information."""
    base, cand = {}, {}
    for arm, sink in ((baseline, base), (candidate, cand)):
        for r in arm.results:
            sink.setdefault(r.item_id, []).append(r.passed)
    shared = sorted(set(base) & set(cand))
    cand_only = sum(
        1 for i in shared
        if majority(cand[i]) and not majority(base[i])
    )
    base_only = sum(
        1 for i in shared
        if majority(base[i]) and not majority(cand[i])
    )
    return cand_only, base_only, len(shared)


def majority(flags: list) -> bool:
    return sum(flags) * 2 > len(flags)


def exact_binomial_two_sided(wins: int, total: int) -> float:
    """P(observing a split at least this lopsided | p = 0.5).
    No SciPy dependency: the sum is small enough to do directly."""
    if total == 0:
        return 1.0

    def pmf(k):
        return math.comb(total, k) * (0.5 ** total)

    observed = pmf(wins)
    return min(1.0, sum(pmf(k) for k in range(total + 1)
                        if pmf(k) <= observed + 1e-12))


def bootstrap_paired_delta(baseline: Arm, candidate: Arm,
                           iters=5000, seed=7) -> tuple:
    """95% interval on the MEAN PAIRED DIFFERENCE in graded score.
    Pairing is what removes task-to-task variance; do not compare
    two independent aggregate means."""
    rng = random.Random(seed)
    b, c = baseline.by_item(), candidate.by_item()
    deltas = [c[i].score - b[i].score for i in sorted(set(b) & set(c))]
    if not deltas:
        return (0.0, 0.0, 0.0)
    means = []
    for _ in range(iters):
        sample = [deltas[rng.randrange(len(deltas))] for _ in deltas]
        means.append(statistics.fmean(sample))
    means.sort()
    lo = means[int(0.025 * iters)]
    hi = means[int(0.975 * iters)]
    return (statistics.fmean(deltas), lo, hi)


def report(baseline: Arm, candidate: Arm) -> dict:
    cand_only, base_only, n = discordant_pairs(baseline, candidate)
    discordant = cand_only + base_only
    p = exact_binomial_two_sided(cand_only, discordant)
    mean_delta, lo, hi = bootstrap_paired_delta(baseline, candidate)
    underpowered = discordant < 25
    return {
        "items_compared": n,
        "candidate_only_wins": cand_only,
        "baseline_only_wins": base_only,
        "discordant_pairs": discordant,
        "binomial_p": round(p, 4),
        "mean_paired_score_delta": round(mean_delta, 4),
        "delta_ci95": (round(lo, 4), round(hi, 4)),
        # Say this out loud rather than reporting a false "no difference".
        "verdict": ("UNDERPOWERED - do not conclude equivalence"
                    if underpowered else
                    "DIFFERENCE" if p < 0.05 else "NO DETECTED DIFFERENCE"),
    }

Two practical notes on running it. First, run both arms in the same window, interleaved rather than one after the other, so that a provider-side capacity fluctuation hits both arms equally. Second, log the raw outputs, not just the verdicts. On day three somebody will ask what the failures actually look like, and re-running to find out costs you half a day you do not have inside the box.

Day 3: decide against a rule set you wrote first

The most important instruction in this guide is procedural rather than technical: write down the accept and reject thresholds on day one, before you have seen a single candidate result, and do not change them on day three.

The reason is uncomfortable but well understood. Once a number is in front of you, you will find a story for it. A four-point drop in success rate becomes "mostly on the hard tasks anyway". A p95 latency increase becomes "within normal variance". A cost saving smaller than expected becomes "still directionally right". None of these reinterpretations are dishonest; they are the ordinary human response to a result you were hoping for, and the only reliable defence is to fix the decision rule while you are still indifferent to the outcome.

Pre-registering also turns the decision into something a second person can audit. A written rule set with a date on it is a document your engineering lead, your finance lead and, in regulated work, your compliance reviewer can all read in two minutes. A verbal judgement formed after the fact is not.

Note how several rows resolve to "exclude that slice" rather than to a global accept or reject. That is deliberate, and it is where day three usually lands.

Partial migration is usually the right answer

The instinct after a good result is to cut everything over. Resist it, because your traffic is almost certainly not uniform. In most production workloads a minority of hard requests consumes a disproportionate share of tokens and carries nearly all of the quality risk, while a large majority of short, well-structured, low-stakes requests carries very little of either. Moving the easy slice captures most of the saving at a fraction of the exposure.

"Easy slice" has to mean something you can evaluate before you call the model, otherwise it is not a routing rule. In practice the usable predicates are: input length below a threshold; task family, where you already have a classifier or an explicit endpoint; expected tool-call depth of zero or one; absence of any regulated, financial or safety-critical flag on the request; a customer tier that is not contractually promised a specific model; and, where your logs support it, historical difficulty for that request shape. If none of those apply, you do not have an easy slice — you have uniform traffic, and a full cutover behind a canary is the right shape instead.

Be clear with yourself that a permanent split is a router, and a router is a system with its own failure modes, its own regression surface and its own evaluation requirements. If you intend to keep the split indefinitely, read our guide to evaluating a router for regret and escalation before you treat it as finished, because the question of whether the cheap model deserved each request is a separate measurement from the one you have just made. For the cutover mechanics themselves — shadow traffic, canary percentages and automatic rollback triggers — our guide to shadow and canary deploys for model upgrades covers the deployment side that this drill deliberately does not.

Common pitfalls that eat the saving

Cache invalidation on cutover. The largest and most predictable one. Long stable prompt prefixes are usually the cheapest tokens on your bill, and switching models resets every cached prefix. Your first days after cutover run at uncached prices while hit rates rebuild, and if your cutover is partial and traffic is split, both models may sit below the hit rate either would achieve alone. Model the transition period explicitly rather than assuming steady state from hour one.

Caching discounts that do not transfer. Subtler and more damaging. Cache write surcharges, cache read discounts, minimum cacheable prefix lengths and cache lifetimes differ by provider and sometimes by model within a provider. A workload engineered around one caching regime can lose more on changed cache economics than it gains on the headline rate. If caching is a material part of your bill, price the candidate's cached path specifically rather than assuming the discount structure carries across; the mechanics are set out in our guide to prompt caching across Claude, GPT and Gemini.

Fast, priority and service-tier premiums. Several providers offer a faster or higher-priority path at a premium over standard pricing, and some SDK defaults or latency-sensitive configurations select it without an obvious signal. Confirm which tier your measurements ran on and which tier production will use. Comparing a standard-tier candidate against a priority-tier incumbent, or the reverse, produces a number that is simply wrong.

Batch and asynchronous discounts. If part of your workload is offline and already benefits from a batch discount on the incumbent, check whether the equivalent exists for the candidate and at what ratio. A cut on synchronous list pricing may leave your batch path unchanged or worse.

Comparing against promotional pricing. A promotional or preview rate is a marketing window, not a price. Build the case on rates the provider has stated are permanent, and if you must use a promotional rate, model the post-promotion price as your downside case. The September 2026 OpenAI cuts described earlier were explicitly confirmed as permanent rather than promotional, which is precisely the property that makes them safe to plan against; not every cut carries that confirmation, and it is worth asking for.

Counting a saving you never bank. The quietest one of all. Cheaper tokens routinely cause volume to grow, because teams stop rationing, agents get more turns, and context windows get filled more generously. That may be an excellent outcome for the product, but it is not a cost reduction, and reporting it as one damages your credibility the next time you ask for a migration slot. Report the unit saving and the total spend separately, always. The broader discipline of proving a cost reduction to a finance audience — the difference between a unit saving and a bill that actually falls — is worked through in our guide to cost-aware evals and scoring quality per pound.

A UK team paying in pounds and an Indian team paying in rupees will each find the currency question layered on top of all of these. A fifteen per cent unit saving denominated in US dollars can disappear entirely against a nine per cent currency move, and a finance lead reviewing the migration six months later sees only the reporting-currency line. State the unit saving in dollars, the reporting-currency effect separately, and the exchange rate and date you used for both.

"""Cost per task from logged token counts.

Reads your own production trace records -- one row per model call -- groups
them into completed tasks, and produces cost per SUCCESSFUL task by family.
Prices are supplied by you, with the date you took them, because an undated
price is the most common way a migration case quietly becomes wrong.
"""
from collections import defaultdict
from dataclasses import dataclass


@dataclass(frozen=True)
class PriceCard:
    """Per-million-token rates. Stamp the date and the source. ALWAYS."""
    model: str
    input_per_mtok: float
    output_per_mtok: float
    cached_input_per_mtok: float   # often a fraction of the input rate
    cache_write_per_mtok: float    # often a premium over the input rate
    currency: str                  # "USD" -- convert at report time, not here
    priced_on: str                 # "2026-09-23"
    source: str                    # provider price page URL
    tier: str = "standard"         # "standard" | "priority" | "batch"

    def call_cost(self, rec: dict) -> float:
        fresh_in = max(0, rec["input_tokens"] - rec.get("cached_input_tokens", 0))
        return (
            fresh_in / 1e6 * self.input_per_mtok
            + rec.get("cached_input_tokens", 0) / 1e6 * self.cached_input_per_mtok
            + rec.get("cache_write_tokens", 0) / 1e6 * self.cache_write_per_mtok
            + rec["output_tokens"] / 1e6 * self.output_per_mtok
        )


def cost_per_task(records, cards, fx_rate=1.0, fx_dated_on=None):
    """records: iterable of dicts with keys
         task_id, family, model, input_tokens, output_tokens,
         cached_input_tokens, cache_write_tokens, succeeded (bool, on the
         final call of the task only), wall_clock_ms
       cards:   {model_name: PriceCard}

    Returns per-family aggregates. Note that the denominator is SUCCESSFUL
    tasks, not calls and not attempts -- a model that fails cheaply is not
    cheap.
    """
    spend = defaultdict(float)
    calls = defaultdict(int)
    tasks = defaultdict(set)
    wins = defaultdict(set)
    wall = defaultdict(list)
    unpriced = set()

    for rec in records:
        card = cards.get(rec["model"])
        if card is None:
            unpriced.add(rec["model"])
            continue
        key = rec["family"]
        spend[key] += card.call_cost(rec)
        calls[key] += 1
        tasks[key].add(rec["task_id"])
        if rec.get("succeeded"):
            wins[key].add(rec["task_id"])
            wall[key].append(rec.get("wall_clock_ms", 0))

    out = {}
    for family in sorted(tasks):
        n_tasks = len(tasks[family])
        n_wins = len(wins[family])
        durations = sorted(wall[family])
        out[family] = {
            "tasks": n_tasks,
            "successes": n_wins,
            "success_rate": round(n_wins / n_tasks, 4) if n_tasks else 0.0,
            "calls_per_task": round(calls[family] / n_tasks, 2) if n_tasks else 0,
            "spend": round(spend[family] * fx_rate, 4),
            # None, not zero: a model that never succeeds has no cost per
            # success, it has a failure to report.
            "cost_per_successful_task": (
                round(spend[family] / n_wins * fx_rate, 6) if n_wins else None
            ),
            "wall_clock_p50_ms": durations[len(durations) // 2] if durations else None,
            "fx_rate": fx_rate,
            "fx_dated_on": fx_dated_on,
        }
    if unpriced:
        out["_warning_unpriced_models"] = sorted(unpriced)
    return out


def compare(baseline_report, candidate_report, min_saving=0.15):
    """Apply the pre-registered cost threshold mechanically. Do not
    adjust min_saving after seeing these numbers."""
    rows = []
    for family, base in baseline_report.items():
        if family.startswith("_"):
            continue
        cand = candidate_report.get(family)
        if not cand or not base["cost_per_successful_task"] \
                or not cand["cost_per_successful_task"]:
            rows.append((family, None, "NO COMPARABLE DATA"))
            continue
        delta = (cand["cost_per_successful_task"]
                 - base["cost_per_successful_task"]) \
            / base["cost_per_successful_task"]
        verdict = "PASSES COST GATE" if delta <= -min_saving else "FAILS COST GATE"
        rows.append((family, round(delta, 4), verdict))
    return rows

After the drill: making the next one take a day

The drill is worth running twice. It is worth running six times if you keep the artefacts, because most of the three days is setup that does not need repeating. Price cuts are now frequent enough — the September 2026 round described above was one of several during the year — that a team which cannot re-qualify quickly is systematically leaving money on the table.

Record seven things, in one dated page, versioned next to the eval set:

• The pre-registered rule set, with the date it was written and who agreed it. Next time you edit it rather than invent it.
• The frozen task set version, with a note on what it does and does not cover, so the next person knows whether it still represents your traffic.
• The dated baseline: cost per successful task, tokens per task, calls per task and p95 latency by family, with the price card and exchange rate used.
• The paired harness configuration: repeats, retry budget, concurrency, grader version and the interleaving scheme.
• The raw outputs and verdicts from both arms, not just the summary statistics. They are the artefact you will actually want in six weeks.
• The slice definition you migrated, expressed as the routing predicate you actually deployed, plus the rollback switch and where it lives.
• The verdict paragraph, one paragraph, dated, naming the gate that decided it — including for a rejection, which is the one people skip and the one that stops the same conversation recurring.

With those in place, the next price cut is a day: re-quote the prices, run the existing harness against the new candidate, apply the existing thresholds, and update the slice boundary. The judgement was made once, in cold blood, and reused.

Where this drill sits among the others

Three neighbouring exercises get confused with this one, and keeping them apart saves a lot of scope creep. Qualifying a model you have never used before is a longer, wider piece of work with licence, shape and infrastructure gates in it. Designing a permanent router is a systems problem, not an evaluation problem. And leaving a vendor under duress — a deprecation notice, a contract ending, an availability change — is a different discipline again, with a different tempo and a much larger blast radius; if that is your actual situation, our guide to writing an LLM vendor exit plan before you need it is the one to read instead.

This drill is narrower than all three, and that narrowness is the point. One candidate, one scaffold, one frozen task set, three days, thresholds written first. The prize is not that you always move. The prize is that within three days you know, in writing, whether you should — and that the decision was made by a rule you set while you still had no stake in the answer.