Evaluating Your Router: Did the Cheap Model Deserve It?

Your router saved 60 per cent. At what quality cost?

There is a sentence that gets said in a lot of engineering reviews, and it is always said with confidence: "we put a router in front of the models and inference spend dropped sixty per cent." It is usually true. It is also, on its own, meaningless — in exactly the way that "we made the database faster by deleting rows" is meaningless. You have described the saving and said nothing about what was given up to obtain it.

The reason this passes unchallenged is structural rather than careless. Cost is measured for you. Every provider hands you a per-request cost, your gateway aggregates it, and the month-on-month drop is legible to anyone who can read a chart. Quality is not measured for you. Nobody sends you an invoice that says "this month you shipped 3.1 per cent worse answers because the small model handled things it should not have." So the visible half of the trade-off gets reported and the invisible half gets assumed away. A router is, structurally, a machine for converting quality into money at a rate nobody has measured.

This guide is not about how to build a router. If you are still choosing between an embedding classifier, a small scoring model and a confidence-threshold cascade, the build-side decisions live in our guide to model routing and cascades as a cost technique, and this piece assumes you have already shipped something along those lines. What follows is the part that almost always gets skipped afterwards: measuring the thing you built, on purpose, with numbers you can defend when someone asks whether the sixty per cent was worth it.

The framing that makes the question answerable is regret. Imagine an oracle router that, for every request, knows in advance which tier will get it right and picks that one. The oracle's quality is the ceiling available to any router working over your tiers. Regret is the gap between that ceiling and what your router actually delivered. It is a small, bounded, comparable number, and — this is the part that matters — it is only interpretable when reported next to the money saved. Two points of regret for a seventy-seven per cent cost reduction is an excellent router. Two points of regret for an eight per cent cost reduction is a bad one wearing the same number. Report one without the other and you have said nothing.

There is also a market reason this matters more in late 2026 than it did a year earlier. Routing has stopped being purely something you build and started being something you buy. Sakana's Fugu Ultra v2, released on 11 September 2026, is not a single model at all — it is a learned orchestrator that routes across a pool of open and specialised models, priced at 5 US dollars per million input tokens and 30 US dollars per million output tokens (pricing as of September 2026). Its cost-optimised sibling, Fugu Max, targets frontier-adjacent quality at two to six times lower cost by dispatching to the leanest model capable of the task. If you adopt a product like that, you have not escaped router evaluation. You have acquired a router whose internals you cannot inspect, which raises the value of measuring it from the outside rather than lowering it.

Every routing decision has four possible outcomes

The conceptual spine of everything below is a two-by-two. On one axis, what your router did: kept the item on the small tier, or escalated it to the large one. On the other axis, the counterfactual truth: whether the small tier would have answered correctly. Four cells, two of which are wins and two of which are failures — and the two failures are not symmetric in any respect that matters.

Missed escalation is the failure mode the routing literature names as the critical one, and the naming is deserved. It is the only cell of the four that is completely invisible from production telemetry. Think about what your logs contain after a cheap-routed request: the prompt, the small model's answer, the latency, the cost. What they do not contain, and cannot contain, is what the large model would have said, because you did not call it. The evidence needed to detect the failure was never generated. Unnecessary escalation, by contrast, announces itself on the invoice at the end of the month, which is precisely why every team has a story about tightening a threshold to cut cost and no team has a story about loosening one to cut missed escalations.

One refinement before the metrics. The right positive class is not "the small tier was wrong" but "escalating is useful" — that is, the large tier is correct where the small tier is not. Items that both tiers get wrong are not escalation opportunities; sending them up buys you a worse invoice and the same wrong answer. And there is a fourth, rarer state worth keeping in view: the item the small tier got right and the large tier gets wrong. It happens more often than people expect, particularly on short classification-shaped tasks where a bigger model overthinks, and it means an unnecessary escalation can be actively harmful rather than merely wasteful.

The metrics that fall out of the two-by-two

Once the four cells are named, the metrics write themselves. Treat "escalation is useful" as the positive class, and the standard confusion-matrix vocabulary applies directly, with one addition — regret — that the confusion matrix does not give you because it carries no notion of what quality was actually delivered.

Note the denominators carefully. Precision and recall are computed over decisions of a given kind; the two rates are computed over all traffic. This is deliberate. Escalation recall of 0.64 sounds survivable in the abstract. The same router expressed as a missed-escalation rate of 4.1 per cent, over eleven million requests a month, is roughly 450,000 requests answered worse than they needed to be, and that is the version of the number that changes behaviour in a review.

For calibration rather than as a target: published cascade systems in the routing literature have reported escalation precision and recall in the low-to-high eighties, with F1 landing between them. Treat any single such triple as an illustrative result and nothing more — we are deliberately not quoting one to two decimal places here, because a precise-looking number from someone else's task distribution invites exactly the false comparison this article is arguing against. They came from that system's task distribution, that system's tier pair and that system's quality judge, and there is no reason at all to expect your numbers to land nearby. The value of the figures is that they tell you what shape a competently measured router looks like when someone reports it honestly: three numbers, together, with the set described.

Here is the arithmetic, written so that it can be dropped into a notebook and checked. Note the assertion at the end — it exists to demonstrate that regret decomposes exactly into missed escalations plus the harmful subset of unnecessary ones, which is a property worth being able to verify rather than take on trust.

from dataclasses import dataclass


@dataclass(frozen=True)
class RoutingDecision:
    """One fully labelled routing decision, from an eval set or shadow log.

    routed_expensive  : did the router send this item to the large tier?
    cheap_correct     : was the small tier's answer judged correct?
    expensive_correct : was the large tier's answer judged correct?
    cheap_cost        : cost of the small-tier call, your currency
    expensive_cost    : cost of the large-tier call, your currency
    """
    routed_expensive: bool
    cheap_correct: bool
    expensive_correct: bool
    cheap_cost: float
    expensive_cost: float


def _safe_div(numerator: float, denominator: float) -> float:
    """Zero rather than ZeroDivisionError — empty cells are normal at
    extreme thresholds (no escalations at all, for instance)."""
    return numerator / denominator if denominator else 0.0


def score_router(decisions: list[RoutingDecision]) -> dict:
    n = len(decisions)
    if n == 0:
        raise ValueError("score_router needs at least one labelled decision")

    # Escalation confusion matrix. The positive class is "escalation is
    # USEFUL": the large tier is right where the small tier is wrong.
    # Items both tiers get right, and items both tiers get wrong, are
    # negatives — escalating them buys no quality.
    justified = 0      # routed expensive, escalation useful      (TP)
    unnecessary = 0    # routed expensive, escalation not useful  (FP)
    missed = 0         # routed cheap,     escalation useful      (FN)
    correct_saving = 0 # routed cheap,     escalation not useful  (TN)

    # Diagnostic split of the unnecessary cell.
    fp_both_correct = 0  # wasted money only
    fp_both_wrong = 0    # wasted money, no quality change either way
    fp_harmful = 0       # wasted money AND lost an answer the small tier had

    served_correct = 0   # quality the router actually delivered
    oracle_correct = 0   # quality a perfect router would have delivered
    served_cost = 0.0
    always_expensive_cost = 0.0
    always_cheap_cost = 0.0

    for d in decisions:
        escalation_useful = d.expensive_correct and not d.cheap_correct

        if d.routed_expensive:
            if escalation_useful:
                justified += 1
            else:
                unnecessary += 1
                if d.cheap_correct and d.expensive_correct:
                    fp_both_correct += 1
                elif d.cheap_correct:          # large tier lost a won item
                    fp_harmful += 1
                else:                          # neither tier was correct
                    fp_both_wrong += 1
            delivered = d.expensive_correct
            served_cost += d.expensive_cost
        else:
            if escalation_useful:
                missed += 1
            else:
                correct_saving += 1
            delivered = d.cheap_correct
            served_cost += d.cheap_cost

        served_correct += int(delivered)
        oracle_correct += int(d.cheap_correct or d.expensive_correct)
        always_expensive_cost += d.expensive_cost
        always_cheap_cost += d.cheap_cost

    # Regret decomposes exactly: every point of lost quality is either an
    # item kept cheap that the large tier would have saved, or an item
    # escalated away from a small tier that already had it right.
    assert oracle_correct - served_correct == missed + fp_harmful

    precision = _safe_div(justified, justified + unnecessary)
    recall = _safe_div(justified, justified + missed)
    f1 = _safe_div(2 * precision * recall, precision + recall)
    saved = always_expensive_cost - served_cost

    return {
        "n": n,
        "escalation_rate": (justified + unnecessary) / n,
        "escalation_precision": precision,
        "escalation_recall": recall,
        "escalation_f1": f1,
        "missed_escalation_rate": missed / n,
        "unnecessary_escalation_rate": unnecessary / n,
        "harmful_escalation_rate": fp_harmful / n,
        "regret": (oracle_correct - served_correct) / n,
        "served_accuracy": served_correct / n,
        "oracle_accuracy": oracle_correct / n,
        "cheap_only_accuracy": None,  # fill from a pinned-cheap run
        "cost": served_cost,
        "cost_saved_vs_always_expensive": saved,
        "savings_rate_vs_always_expensive":
            _safe_div(saved, always_expensive_cost),
        "cost_premium_vs_always_cheap": served_cost - always_cheap_cost,
        "cells": {
            "correct_saving": correct_saving,
            "justified": justified,
            "missed": missed,
            "unnecessary": unnecessary,
            "unnecessary_both_correct": fp_both_correct,
            "unnecessary_both_wrong": fp_both_wrong,
            "unnecessary_harmful": fp_harmful,
        },
    }

Building an eval set where the tiers actually disagree

This is the practical heart of the exercise, and it is where most router evaluations quietly fail before they produce a single number.

The obvious move is to take a random sample of production traffic, label it, and measure the router on it. It is the wrong move, and the reason is worth internalising: most of your traffic is easy. On an easy item, the small tier and the large tier both get it right, so the router's decision was irrelevant — any choice would have produced the same quality, and the only thing at stake was a fraction of a penny. If eighty-five per cent of your sample is like that, then eighty-five per cent of your measured routing accuracy is measuring nothing. Your router will score beautifully. It will score beautifully even if it is a coin flip. A random-sample evaluation of a router is close to a test that cannot fail, which is exactly why it is the one teams run.

The eval set you need is built out of disagreements: items where the tiers produce materially different quality, because those are the only items where the routing decision had any consequence. The construction is mechanical:

One — harvest. Take a large raw sample of real production requests, sized generously; several thousand is a reasonable starting point for a mature product. Replay every one of them through both tiers. This is a deliberate, budgeted spend, and it is the single most useful thing you will buy all quarter. Keep the prompts, both completions, both costs and both latencies.

Two — score both sides. Apply your quality judgement to each pair. If you have deterministic ground truth — a classification label, an extracted field, a passing test — use it, because a deterministic signal is worth ten judged ones. If you do not, this is where the offline judge earns its keep, and you have to accept that the judge is now part of your measuring instrument. The same discipline that makes a verification pass trustworthy applies here; our guide on using a second model to verify and critique covers the mechanics of building one that is not simply an agreement machine.

Three — keep the disagreements. Retain items where the two tiers differ in judged quality. Discard the both-correct items and keep a small stratum of the both-wrong ones — they are not escalation opportunities, but you need enough of them for your unnecessary-escalation accounting to be right, and they are a useful early-warning signal of tasks your system simply cannot do at any tier.

Four — stratify. The disagreement pool is not the eval set until it has been balanced across the dimensions that plausibly move routing behaviour: task type, input length band, language, customer segment, channel, and — for a dual-market product — region. An India-facing support flow and a UK-facing one do not disagree in the same places, because the inputs differ in length, register and code-mixing, and a router tuned on one will systematically misroute the other. If you serve both Mumbai and London, both belong in the set with enough items to compute a per-region regret, or you have measured a router you do not actually run.

Five — label and adjudicate. Two labellers on a sample, an agreement rate you record, an adjudication pass on the disagreements between labellers. If human labelling at that scale is out of reach, label a few hundred by hand, and use them to calibrate the judge that labels the rest — but publish the judge's agreement rate with the humans alongside every number the judge produced.

On sizing, the binomial arithmetic is unromantic and genuinely useful. To pin a proportion near one half — which is the worst case for variance, and roughly where escalation precision sits for many routers — to within five percentage points at ninety-five per cent confidence takes about 384 observations of that kind. To within three points, about 1,067. So if you want escalation precision to plus or minus five points, you need around 384 escalated items in the set, not 384 items overall. Work backwards: at an eighteen per cent escalation rate that implies roughly 2,100 eval items, and if only about twelve per cent of raw traffic yields a tier disagreement, harvesting 2,100 disagreement-bearing items means replaying roughly 17,800 raw requests through both tiers. That is the real cost of a defensible number, and it is far better to know it in advance than to discover it after you have built a set of 200 items that cannot distinguish a good router from a bad one.

Refresh the set on model events, not on a calendar. A new version of either tier invalidates every counterfactual label in the file, because the label was a statement about a model that no longer exists. A prompt change does the same, more quietly. So does a provider silently altering default reasoning behaviour — the kind of change that moves what a model does with its thinking budget across an upgrade can invert a tier's behaviour on exactly the hard items your disagreement set is made of. Treat the eval set as versioned data with an explicit tier pair stamped on it, and re-harvest whenever either half of the pair moves.

Instrumenting the router in production

The offline set tells you how the router behaves on a curated distribution. Production tells you what the distribution actually is. You need both, and the production side requires logging that most teams have not put in, because the interesting fields are the ones about the decision rather than about the call.

Shadow-mode sampling is the only honest way to estimate counterfactual quality on live traffic. The mechanism is simple: for a small random fraction of requests, after serving the user from the chosen tier, also call the tier you did not choose, off the critical path, and log both outputs for later judgement. It is the production analogue of the disagreement harvest, and it has one property the offline set cannot have — it samples the distribution you actually get, including the parts of it that nobody thought to put in an eval set.

Two design points carry most of the value. First, the shadow call must never be able to affect the served response or the user's latency; it runs after the response is returned, in a pool, with failures swallowed and counted. Second, the sampling should be asymmetric. Shadowing the large tier behind cheap-routed traffic is the expensive direction and also the one that finds missed escalations; shadowing the small tier behind escalated traffic is cheap and finds unnecessary escalations. Since unnecessary escalations already show up on the invoice and missed ones show up nowhere, most teams should spend the larger share of their shadow budget on the expensive direction, unintuitive as that feels when someone is reviewing the bill.

import random
import time
import uuid
from concurrent.futures import ThreadPoolExecutor


class ShadowSampledRouter:
    """Serves normally; for a sampled fraction also calls the tier it did
    NOT pick, off the user's critical path, and logs both for labelling."""

    def __init__(self, score_fn, tiers, threshold, sink,
                 shadow_rate_cheap_routed=0.05,
                 shadow_rate_escalated=0.02,
                 rng=None, pool=None):
        self.score_fn = score_fn          # request -> float in [0, 1]
        self.tiers = tiers                # {"cheap": fn, "expensive": fn}
        self.threshold = threshold
        self.sink = sink                  # .write(dict)
        # Asymmetric on purpose: shadowing the LARGE tier behind cheap
        # traffic is the only way to see missed escalations, and missed
        # escalations never appear on the invoice.
        self.shadow_rate_cheap_routed = shadow_rate_cheap_routed
        self.shadow_rate_escalated = shadow_rate_escalated
        self.rng = rng or random.Random()
        self.pool = pool or ThreadPoolExecutor(max_workers=4)

    def handle(self, request, slice_keys=None):
        trace_id = uuid.uuid4().hex
        score = self.score_fn(request)
        escalated = score >= self.threshold
        served_tier = "expensive" if escalated else "cheap"
        other_tier = "cheap" if escalated else "expensive"

        started = time.perf_counter()
        served = self.tiers[served_tier](request)
        latency_ms = (time.perf_counter() - started) * 1000.0

        self.sink.write({
            "kind": "served",
            "trace_id": trace_id,
            "router_version": getattr(self.score_fn, "version", "unknown"),
            "threshold": self.threshold,
            "router_score": score,
            "served_tier": served_tier,
            "served_model_id": served.model_id,
            "output": served.text,
            "cost": served.cost,
            "latency_ms": latency_ms,
            **(slice_keys or {}),
        })

        rate = (self.shadow_rate_escalated if escalated
                else self.shadow_rate_cheap_routed)
        if self.rng.random() < rate:
            # Fire and forget. The user has already been answered.
            self.pool.submit(self._shadow, trace_id, other_tier, request)

        return served

    def _shadow(self, trace_id, tier, request):
        try:
            started = time.perf_counter()
            result = self.tiers[tier](request)
            self.sink.write({
                "kind": "shadow",
                "trace_id": trace_id,
                "shadow_tier": tier,
                "shadow_model_id": result.model_id,
                "shadow_output": result.text,
                "shadow_cost": result.cost,
                "shadow_latency_ms": (time.perf_counter() - started) * 1000.0,
            })
        except Exception as exc:                      # never break serving
            self.sink.write({
                "kind": "shadow_error",
                "trace_id": trace_id,
                "shadow_tier": tier,
                "error": type(exc).__name__,
            })

The cost of this is real and should be budgeted explicitly rather than absorbed. Shadowing cheap-routed traffic at five per cent adds five per cent of the large-tier price on that slice of traffic — which, given that price gaps between tiers routinely run an order of magnitude, is not a rounding error. As of September 2026, DeepSeek's V4.1 Flash was priced at 0.30 US dollars per million input tokens and 1.20 per million output at peak, dropping to 0.15 and 0.60 off-peak (pricing as of September 2026), against frontier tiers an order of magnitude higher. That gap is the whole reason routing is worth doing, and it is also the reason shadow traffic in the expensive direction has to be sized deliberately. Fold it into the same model you use to forecast an LLM bill before launch, as an evaluation line item sitting beside inference rather than hidden inside it. One practical lever: shadow more heavily off-peak where a provider offers off-peak pricing, since a shadow call has no user waiting on it and can be deferred by hours.

The threshold is the product

Here is the thing that surprises people who have just built a router: the classifier is rarely the interesting part. The interesting part is the threshold, because the threshold is where your organisation's actual appetite for the quality-versus-cost trade is encoded, and in most teams it was set by someone typing a round number and moving on.

The disciplined alternative is to sweep it. Take your scored eval set, evaluate every candidate threshold, and produce a table of escalation rate, precision, recall, regret and projected cost. Then pick a point on that curve against a stated budget or a stated regret ceiling — a decision an engineering manager can make in five minutes with the table in front of them and cannot make at all without it.

from dataclasses import dataclass


@dataclass(frozen=True)
class ScoredItem:
    """An eval-set item with the router's score and BOTH tiers labelled."""
    score: float
    cheap_correct: bool
    expensive_correct: bool
    cheap_cost: float
    expensive_cost: float


def sweep_thresholds(items, thresholds, monthly_requests=1_000_000):
    """Re-decide every item at each candidate threshold and re-score."""
    rows = []
    for t in sorted(thresholds):
        decisions = [
            RoutingDecision(
                routed_expensive=(it.score >= t),
                cheap_correct=it.cheap_correct,
                expensive_correct=it.expensive_correct,
                cheap_cost=it.cheap_cost,
                expensive_cost=it.expensive_cost,
            )
            for it in items
        ]
        m = score_router(decisions)
        rows.append({
            "threshold": t,
            "escalation_rate": m["escalation_rate"],
            "precision": m["escalation_precision"],
            "recall": m["escalation_recall"],
            "missed_escalation_rate": m["missed_escalation_rate"],
            "regret": m["regret"],
            "served_accuracy": m["served_accuracy"],
            "cost_per_request": m["cost"] / m["n"],
            "projected_monthly_cost":
                (m["cost"] / m["n"]) * monthly_requests,
            "missed_per_month":
                m["missed_escalation_rate"] * monthly_requests,
        })
    return rows


def pick_threshold(rows, monthly_budget=None, max_regret=None):
    """Lowest regret inside the budget; on a tie prefer the higher
    threshold, because a higher threshold is the cheaper one."""
    candidates = rows
    if monthly_budget is not None:
        candidates = [r for r in candidates
                      if r["projected_monthly_cost"] <= monthly_budget]
    if max_regret is not None:
        candidates = [r for r in candidates if r["regret"] <= max_regret]
    if not candidates:
        raise ValueError(
            "no threshold satisfies the constraints — either the budget is "
            "below the cheap tier's own cost, or the regret ceiling is "
            "below what this tier pair can deliver at any threshold"
        )
    if monthly_budget is not None:
        return min(candidates, key=lambda r: (r["regret"], -r["threshold"]))
    return min(candidates, key=lambda r: (r["projected_monthly_cost"],
                                          r["regret"]))

The output of that sweep is the single most useful artefact in router evaluation. Here is a worked one. The figures are illustrative — constructed to be internally consistent rather than measured on any real system — for a tier pair costing 0.40 US dollars per thousand requests on the small tier and 6.00 on the large one, over a set in which 11.2 per cent of items genuinely benefit from escalation and 1.5 per cent are actively harmed by it.

Three things in that table repay a slow read. First, the row at 0.20 has lower regret than escalating everything. That is not a mistake: escalating everything incurs the harmful-escalation regret in full, and a router that declines to escalate a few items the small tier had already won can beat the always-expensive baseline on quality as well as on cost. Second, the interval between 0.35 and 0.65 is where the argument actually lives — sixteen percentage points of extra saving bought with 2.7 additional points of regret. Whether that is a good trade is a product question about what a wrong answer costs in your domain, not an engineering question, and the table is what lets you put it to the person who should answer it. Third, notice that precision rises monotonically as the threshold tightens while recall collapses. A router reported with precision alone looks best exactly where it is doing least.

The dual-market wrinkle is that this curve is not the same in every region you serve, and not only because traffic differs. Endpoint availability differs: a model tier available in a London region may not be available in a Mumbai region, or may be available only at a different version, which forces a different tier pair and therefore a different curve. Latency budgets differ too. Escalation usually costs seconds, and what those seconds cost depends on the product — a UK-facing interactive assistant where the large tier adds three seconds to a p95 may find escalation more expensive in user-experience terms than in rupees or pounds, while an India-facing asynchronous workflow may barely notice. Sweep and pick per region. A single global threshold is a decision to be wrong in at least one market.

Escalate on value, not only on uncertainty

A research direction worth tracking, and worth prototyping if your escalation budget is tight: weighting the escalation decision by the value of being right rather than by model uncertainty alone. The intuition is that uncertainty is the wrong signal by itself, because it is symmetric — the model is equally unsure about the trivial query and the one that determines whether a customer's refund is processed. Escalate when being wrong is expensive, not merely when the model is unsure, and precision improves without giving up recall.

Related published work goes further, framing a signed routing score that separately predicts when escalation will help and when it will hurt, and ranking requests by the difference between those two under a fixed escalation budget. That framing maps neatly onto the two-by-two above — it is, in effect, learning the justified cell and the harmful cell as distinct targets rather than collapsing them into one confidence number. Present this to your own team as what it is: an active research direction rather than settled practice with a reference implementation waiting for you. If you want to follow it, the relevant literature sits under the survey work on dynamic model routing and cascading and the decision-theoretic treatments of when cascades pay off — search those terms rather than taking our summary as a citation. It is, however, the direction in which the interesting routers are heading, and a value-weighted variant is straightforward to try once you already have the labelled disagreement set this article has been building towards.

The failure modes you only see over time

A router evaluated once is a router evaluated on a distribution that has since moved. Four decay modes account for most of the damage, and all four are slow, which is what makes them dangerous — nothing pages anyone, the numbers drift by fractions, and one day the savings chart is still beautiful and the product is worse.

Distribution drift makes the router stale. Your router learned a boundary on the traffic you had. Traffic changes: a new customer segment onboards, a feature ships that produces longer inputs, a marketing campaign brings in users who ask differently. The boundary does not move with it. The symptom is usually that escalation rate stays flat while missed-escalation rate climbs, which is the worst possible combination to detect, because a flat escalation rate is exactly what a cost dashboard is built to reassure you about. Monitor missed-escalation rate from shadow traffic as a first-class metric with an alert on it, not as a quarterly analysis.

A model upgrade silently inverts the tier ordering. This one is genuinely common in late 2026, because small models are improving faster than large ones on several task families. If the small tier is upgraded and becomes better than the large tier on, say, structured extraction, then every escalation on that task family is now a harmful escalation — you are paying more for worse answers, with total confidence, because the tier labels in your code still say what they said six months ago. Cost dashboards will show this as increased spend with stable quality and nobody will investigate. The defence is to re-run the disagreement harvest on every version bump of either tier and to compare the two tiers' accuracy per task family before touching the threshold. Related: if your tiers span providers, the portability work described in keeping structured outputs portable across a provider swap is what makes re-running that harvest a day's work instead of a fortnight's.

The quality judge drifts. If your labels come from an LLM judge and the judge's underlying model is updated, every historical number in your series changes meaning, and your regret trend becomes a chart of judge behaviour rather than router behaviour. Pin the judge's model version, stamp judge_version on every label, and keep a small frozen set of human-labelled items that you re-run through the judge on every version to detect the shift. If judge agreement with the frozen set moves by more than a couple of points, your metric series has a discontinuity in it and should be annotated as such.

The router was trained against a biased judge. The deepest of the four, and the one nobody wants to hear. A router cannot be better than the quality signal it was tuned against. If you tuned on a judge that systematically prefers longer, more hedged answers — a well-documented tendency — you have trained a router that escalates whenever brevity is appropriate, and every metric in this article will confirm that it is doing a wonderful job, because the metrics and the router share a judge. The only defence is to validate the judge against something outside itself: human labels, deterministic ground truth where any exists, or downstream business outcomes. The reliability-harness discipline in our guide on setting a reliability target and qualifying against a harness is the same idea applied to agents — decide what "correct" means before you build the machine that optimises for it, because afterwards is too late.

Evaluating a router you did not build

Increasingly the router is somebody else's. A vendor orchestrator dispatches across a pool of models by criteria it does not disclose, and you get a single endpoint, a single price and a promise. The evaluation problem does not disappear; it changes shape, because you have lost the ability to inspect the decision and kept full responsibility for its consequences.

What to demand from the vendor, and treat the absence of any of it as a finding rather than an inconvenience:

Per-request model disclosure. Which model actually served this request, returned in the response or the usage record. Without it you cannot attribute a quality regression to the routing policy rather than to a model, and you cannot detect the tier-inversion failure at all.

A pinning escape hatch. The ability to force a specific model, or to disable routing entirely, on a per-request basis. This is the single most important capability, because it is what lets you construct the counterfactual: run the same eval set pinned-cheap, pinned-expensive and routed, and you can compute regret and savings for the routed configuration against both poles, exactly as in the code above. No pinning means no counterfactual means no evaluation, only vibes and an invoice.

Change notification on the routing policy. A vendor's policy can change without any version number you can see. Ask what the notification commitment is and what the rollback path looks like. If the answer is that policies improve continuously and silently, you have learned that your quality baseline is not under your control and you should be re-measuring far more often.

Cost attribution granular enough to slice. Aggregate spend is not enough. You need cost per request joined to your own slice keys, or you cannot tell whether the orchestrator is quietly routing your Indian traffic differently from your UK traffic — which it may well be doing for entirely legitimate regional-availability reasons, with quality consequences that are yours to discover.

What you can measure from outside, regardless of what the vendor provides: end-to-end quality on your own held-out set, cost per successful task rather than cost per token, and the variance of both over time. Cost per successful task is the metric that survives contact with orchestration, because it collapses the routing policy, the retry behaviour and the model mix into one number that tracks the thing you actually buy — and the shift towards reporting cost per task rather than per million tokens is visible across the market now, including in how index scores are increasingly quoted on a cost-per-task basis. Run your disagreement set against the vendor endpoint monthly, plot cost per successful task and regret against your own pinned baselines, and you will detect a policy change within a cycle even if nobody told you about one.

If you take one thing from all of this, make it the smallest version: pick two hundred production requests from your next working week, run both of your tiers over all of them, label the disagreements by hand, and compute the four cells. It is a day of work and roughly the cost of a long lunch. Most teams who do it discover that the cell they were most confident about — correct savings — is smaller than they assumed, and that the cell they had never named at all is where their product has been quietly losing. That is the whole argument for measuring the router: not that routing is a bad idea, but that a trade nobody has priced is not a trade, it is a hope.