Model Routing and Cascades: Cut LLM Spend Without Losing Quality

Why routing is the biggest lever you have

Every cost-reduction technique for LLM workloads sits somewhere on a spectrum of effort against return. Prompt caching is cheap to enable and saves on repeated prefixes. Compression trims tokens at the margin — we covered the mechanics in our guide to prompt compression in production. Batch processing halves the price of anything that can wait. All worth doing. But none of them touches the single largest line in most inference budgets, which is this: the price gap between model tiers is a multiple, not a percentage, and most teams send everything to the top tier.

As of August 2026, Claude Haiku 4.5 costs $1 per million input tokens and Claude Opus 5 costs $5 — with output at $5 against $25. That is a five-times spread inside one vendor's line-up, before you look across vendors, where the spread runs to twenty-five times or more. If 60 or 70 per cent of your traffic is the kind of request a small model answers perfectly well — classification, short extraction, formulaic summaries, FAQ-shaped support queries — then routing that majority to the cheap tier is worth more than every other optimisation on your list combined. It is also one of the few levers that improves as your product grows, because volume makes the routing decision better-informed rather than harder.

The research record backs the intuition, with numbers that sound implausible until you read how they are measured. The RouteLLM team at Berkeley reports cost reductions of over 85 per cent on MT-Bench while retaining 95 per cent of GPT-4's performance, and FrugalGPT reports matching the best individual model with up to 98 per cent cost reduction on its evaluation tasks. Both are benchmark results, and your production traffic is messier than a benchmark. But even the deflated real-world version of those numbers — 40 to 70 per cent in the worked example below — dwarfs what any prompt-level tweak can deliver.

The catch, and the reason this guide spends as much time on evaluation and monitoring as on routing itself, is that routing moves quality risk from the vendor's side of the ledger to yours. Sent everything to the flagship, you knew what you were getting. Route it, and the question "is this answer good enough?" becomes your system's job to answer, on every request, forever. Done carelessly, routing is how you find out via customer complaints that your small model has been confidently wrong for a month. Done properly, it is the difference between an AI product with software-adjacent margins and one that loses money on its busiest customers — a topic we treated in depth in LLM unit economics.

The three routing architectures

Every production routing system we have seen is one of three shapes, or a hybrid of them.

The confidence-based cascade is the simplest. Every request goes to the small model first. The small model's answer is checked — by its own self-reported confidence, by a validator, or by both — and if the check fails, the request escalates to the next model up the ladder. Nothing needs training. The strong model backstops every weakness of the weak one, so the failure mode is "paid slightly more", not "shipped a bad answer". The cost is that escalated requests are paid for twice, and the small model's latency is added to every hard request.

The pre-inference classifier puts a decision before the first model call: a lightweight classifier — logistic regression over embeddings, a fine-tuned small transformer, or in the simplest version a rules engine on request metadata — predicts which tier a request needs, and the request is served exactly once by that tier. This is what RouteLLM formalises: train a router on preference data to predict whether the weak model's answer would be acceptable. No double payment, no added latency, but you need labelled data to train it and it can misroute in both directions — sending hard requests down (a quality problem) and easy requests up (a cost problem).

The embedding-based router is the classifier's non-parametric cousin. You maintain a store of past requests embedded as vectors, each labelled with the cheapest tier that handled it acceptably. A new request is embedded, its nearest neighbours are looked up, and it goes wherever similar requests succeeded before. It adapts continuously as new labels arrive, needs no training runs, and doubles as a semantic cache if the neighbour is close enough to reuse outright. Its weakness is novelty: a request unlike anything in the store gets a low-evidence decision, so you need a distance threshold below which you default to a safe tier.

Architecture	Training data needed	Cost per request	Latency on hard requests	Failure mode	When it wins
Confidence cascade	None	Escalations paid twice	Adds small-model attempt first	Over-spends, fails safe	First deployment; low escalation rates; quality-critical products
Pre-inference classifier	Labelled traffic (thousands of examples)	Each request served once	Milliseconds of classifier overhead	Misroutes silently in both directions	High volume; stable traffic mix; team can maintain a model
Embedding router	Labelled store, grows online	Served once + embedding lookup	Single vector search (~10 ms)	Weak on novel request types	Repetitive traffic; doubles as semantic cache; drifting workloads

The pragmatic sequence is to start with the cascade, log everything, and let the logs become the training data for a classifier or an embedding store later. The cascade's escalation decisions are labels: every request that Haiku answered acceptably is a "route low" example, and every escalation is a "route high" example. Six weeks of production traffic through a cascade usually yields a better router training set than anything you could construct up front.

Build a v1 cascade in an afternoon

Here is a minimal but production-shaped cascade over the current Claude ladder. The small model answers and appends a self-assessed confidence score; if the score is below the rung's threshold — or unparseable, which we deliberately treat as zero — the request climbs. The top rung's threshold is zero, so it always accepts.

import re
import anthropic

client = anthropic.Anthropic()

# (model, minimum confidence to accept its answer)
# Pricing as of August 2026: $/MTok input-output
LADDER = [
    ("claude-haiku-4-5-20251001", 80),  # $1 / $5
    ("claude-sonnet-5", 70),            # $3 / $15
    ("claude-opus-5", 0),               # $5 / $25 -- always accepts
]

SUFFIX = (
    "\n\nEnd your reply with one final line in exactly this format:\n"
    "CONFIDENCE: <integer 0-100>\n"
    "Score how confident you are that your answer is correct and complete."
)

def ask(model, prompt):
    resp = client.messages.create(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt + SUFFIX}],
    )
    text = "".join(b.text for b in resp.content if b.type == "text")
    match = re.search(r"CONFIDENCE:\s*(\d{1,3})\s*$", text.strip())
    if match:
        confidence = min(int(match.group(1)), 100)
        answer = text[: match.start()].strip()
    else:
        confidence = 0          # unparseable confidence -> escalate
        answer = text.strip()
    return answer, confidence, resp.usage

def cascade(prompt):
    attempts = []
    for model, threshold in LADDER:
        answer, confidence, usage = ask(model, prompt)
        attempts.append({"model": model, "confidence": confidence,
                         "in": usage.input_tokens, "out": usage.output_tokens})
        if confidence >= threshold:
            return {"answer": answer, "served_by": model,
                    "attempts": attempts}
    # Unreachable: the top rung's threshold is 0, so it always returns above.
    raise RuntimeError("cascade fell through the ladder")

Three design decisions in that snippet matter more than they look. Unparseable confidence escalates rather than accepts — the failure mode of a malformed answer should never be "ship it". The attempts list is returned with the answer, because the record of what each rung cost and claimed is your future training data and your current monitoring feed. And the thresholds are per-rung constants at the top of the file, not buried in logic, because you will tune them weekly for the first month.

Self-reported confidence is a crude signal — models are systematically overconfident on some task families — and the standard upgrade is to pair it with a cheap deterministic validator wherever one exists: does the JSON parse, does the SQL execute against a shadow schema, does the extracted date actually appear in the source document, does the answer cite a retrieved passage. A validator failure forces escalation regardless of the confidence score. In our experience the pair of weak signals comfortably outperforms either alone, and both together still cost nothing compared to a single flagship call.

Measuring "capable enough" before you flip the switch

The cascade above is safe to deploy dark — routing decisions logged but everything still served by the flagship — from day one. What it is not, yet, is safe to trust, because you do not know how often the small model's accepted answers are actually acceptable. The instrument that tells you is a routing eval, and it is built from your production traffic, not from a public benchmark. A leaderboard tells you how models compare on somebody else's distribution; your router only ever sees yours.

The construction is mechanical. Sample a few hundred real prompts from recent production traffic, stratified by route or task family so the rare-but-hard cases are represented, and scrubbed of anything personal. Generate a reference answer for each with the model at the top of your ladder. Then run each candidate small model over the same prompts and grade its answers against the references — with a deterministic check where the task allows one, and an LLM judge where it does not.

import json
import random

def build_routing_eval(prompts, k=500):
    sample = random.sample(prompts, k)
    rows = []
    for p in sample:
        reference, _, _ = ask("claude-opus-5", p["text"])
        candidate, conf, _ = ask("claude-haiku-4-5-20251001", p["text"])
        verdict = judge(p["text"], reference, candidate)  # LLM-as-judge
        rows.append({
            "prompt_id": p["id"], "task_family": p["family"],
            "candidate_confidence": conf,
            "acceptable": verdict.acceptable,   # bool
        })
    return rows

def report(rows):
    by_family = {}
    for r in rows:
        by_family.setdefault(r["task_family"], []).append(r)
    for family, group in sorted(by_family.items()):
        ok = sum(r["acceptable"] for r in group) / len(group)
        print(f"{family:30s} small-model acceptable: {ok:.0%}")

The output you are looking for is a per-family table: on which slices of traffic does the small model clear the bar 90-plus per cent of the time, and on which does it collapse. That table sets your initial thresholds, tells you which routes to exclude from routing entirely, and — critically — gives you the number the CFO will ask for: the percentage of traffic that can move down-tier at measured quality. It also calibrates the confidence signal itself: plot self-reported confidence against judged acceptability and you will find the confidence level below which answers are genuinely unreliable, which is where the threshold belongs, rather than at a round number you guessed.

Two disciplines keep the eval honest over time. First, it is a living artefact: refresh the sample quarterly, because your traffic mix drifts and last winter's distribution is not this summer's. Second, hold out a slice that is never used for threshold tuning, so you always have an untouched measure of how the tuned system generalises. If you later port the same suite across vendors' models, the ground rules in our guide to porting a prompt suite across Claude, GPT and Gemini apply directly — the eval is the portable asset, the prompts are not.

The maths at one million requests a month

Numbers make the case better than argument. Take a workload of one million requests a month, averaging 2,000 input tokens and 500 output tokens per request — a realistic shape for a RAG-backed assistant. Using Claude list pricing as of August 2026 (Haiku 4.5 at $1/$5 per million input/output tokens, Sonnet 5 at $3/$15, Opus 5 at $5/$25 — Sonnet 5 carries an introductory $2/$10 rate through 31 August 2026, which we ignore here as temporary), the monthly bill under four serving strategies:

Strategy	Traffic split	Monthly cost	Saving vs all-Opus
Everything on Opus 5	100% Opus	$22,500	—
Everything on Sonnet 5	100% Sonnet	$13,500	40%
Cascade (Haiku first, 30% escalate to Sonnet, 5% on to Opus)	All hit Haiku; 30% re-served	$9,675	57%
Classifier router (served once per request)	70% Haiku / 25% Sonnet / 5% Opus	$7,650	66%

The arithmetic behind the cascade row: every request pays the Haiku attempt ($4,500 across the month), the 30 per cent that escalate pay Sonnet again ($4,050), and the 5 per cent that climb to the top pay Opus as well ($1,125). Even paying for escalations twice, the cascade takes 57 per cent off the flagship-only bill; remove the double payment with a classifier and the saving reaches 66 per cent. That 40–70 per cent band is what the worked example brackets, and it is worth noticing that the single biggest jump in the table — 40 per cent — comes from the least sophisticated move available, simply not defaulting to the largest model.

Those published results deserve their citations and their caveats in the same breath. RouteLLM (Ong et al., 2024) reports cost reductions of over 85 per cent on MT-Bench, 45 per cent on MMLU and 35 per cent on GSM8K versus GPT-4-only, while retaining 95 per cent of GPT-4 performance — on those benchmarks, with routers trained on preference data. FrugalGPT (Chen, Zaharia and Zou, 2023) reports matching the best individual LLM with up to 98 per cent cost reduction on its evaluation tasks using a learned cascade. Both are honest reported results; neither is a promise about your traffic. Treat them as an upper bound on what a well-tuned router can do when the traffic mix is favourable.

Two forces make this lever more valuable over time, not less. Small models keep absorbing capabilities that needed a flagship a year earlier, so the acceptable-at-the-bottom share of your traffic drifts upward on its own. And vendors keep repricing the cheap tiers aggressively — OpenAI's budget models now start at $0.20 per million input tokens after the Luna price cut, and Google's Gemini line runs from roughly $0.10 input at the Flash-Lite end to flagship rates at the top (vendor-published figures as of August 2026; check current price pages before modelling). A routing layer is the piece of infrastructure that lets you bank each of those improvements the week it ships, rather than after a migration project. Current Claude rates are at anthropic.com/pricing.

Policy overlays: tier, SLA and budget caps

Once the routing layer exists, it quietly becomes the most useful policy enforcement point in your stack, because it is the one place every model call already passes through. Three overlays earn their keep almost immediately.

Customer tier. Enterprise contracts that promise "our best model" can pin those tenants' traffic to the top of the ladder, or set their escalation thresholds so low that most requests climb; free tiers can cap at the bottom rung with no escalation at all. This turns model quality from an engineering accident into a priced product attribute — the difference between plans stops being a usage cap nobody hits and becomes something customers can feel.

Latency and SLA. Cascades add the small model's latency to every escalated request, which is fine for asynchronous work and painful for interactive surfaces with a tight latency budget. Route interactive traffic through a classifier (one serve, milliseconds of overhead) and let batch and background traffic take the cascade, where the double-serve costs nothing anyone can perceive. If you commit to regional processing — inference pinned to AWS Mumbai for Indian data-residency requirements, or London for UK clients — the routing layer is also where that pin lives, since region and tier are decided at the same moment.

Budget caps. A router that knows the running spend per tenant can degrade gracefully as a cap approaches: tighten thresholds, drop the ceiling from Opus to Sonnet, queue non-urgent work for batch pricing — rather than the two bad alternatives of a hard cut-off or a blown budget. For products billing in rupees or pounds against dollar-denominated inference, this is also the natural place to absorb exchange-rate movement before it reaches gross margin.

Keep the policy table in configuration, not code — per-tenant and per-route entries for floor model, ceiling model, thresholds and caps. Every one of these decisions will be revisited by someone who is not an engineer, and the cheapest way to support that is to make the policy legible and editable without a deployment.

Failure modes: silent regression and escalation storms

Routing systems fail in two characteristic directions, and both are quiet by default.

Silent quality regression is the insidious one. The small model's acceptable-answer rate drifts downward — a prompt change that interacts badly with the small model, a shift in traffic mix towards families it handles poorly, a model version update under an alias — while the confidence signal keeps reporting the same cheerful numbers, because miscalibration is precisely the failure. Nothing errors. The dashboard is green. Quality has regressed for the slice of users whose requests stopped escalating, and the first external signal is churn, weeks later.

The defence is shadow sampling: continuously pick a small fraction of small-model-served requests — one to five per cent — and have the flagship (or, on a slower loop, a human) grade the answer asynchronously. The grade never touches the user-facing response; it feeds a single trended metric, the shadow agreement rate. That number moving is your early warning, and it moves weeks before complaint volume does. Alert on the trend, not the level.

Escalation storms are the loud failure wearing camouflage. Some change — a deploy that mangles the confidence suffix, a new traffic source, a validator misconfigured to fail everything — pushes the escalation rate from 30 per cent to 90, and your cascade silently becomes "flagship plus a Haiku tax on every request": the worst of both architectures. Because everything still returns correct answers, nothing pages; you find out from the invoice. The mirror-image failure is an escalation rate that collapses towards zero, which is almost never good news either — it usually means the confidence parse is broken in the permissive direction and everything is being accepted.

-- Escalation rate per hour, with a 7-day same-hour baseline.
-- Alert when the ratio leaves the band in either direction.
WITH hourly AS (
  SELECT date_trunc('hour', ts)                        AS hr,
         AVG(CASE WHEN served_by <> ladder_bottom
                  THEN 1.0 ELSE 0.0 END)               AS esc_rate
  FROM llm_requests
  GROUP BY 1
)
SELECT hr, esc_rate,
       AVG(esc_rate) OVER (
         ORDER BY hr ROWS BETWEEN 168 PRECEDING AND 1 PRECEDING
       )                                               AS baseline_7d,
       esc_rate / NULLIF(AVG(esc_rate) OVER (
         ORDER BY hr ROWS BETWEEN 168 PRECEDING AND 1 PRECEDING
       ), 0)                                           AS vs_baseline
FROM hourly
ORDER BY hr DESC;

The minimum monitoring set for a routed system is therefore four trended numbers: escalation rate (both directions), shadow agreement rate, cost per served request, and the confidence-parse failure rate. All four come free from the attempts log the cascade already writes. A team in Bengaluru or Birmingham can stand the whole set up in a day on whatever warehouse they already run — the hard part is deciding to treat the router as a product surface with an owner, not a config file nobody revisits.

Where to start

The sequence that gets a team from nothing to a defensible routed system is short. Instrument first: make sure every model call is logged with tokens, model and outcome, because nothing downstream works without it. Build the routing eval from two weeks of production prompts and find out, per task family, what the small model can actually carry. Ship the confidence cascade dark — decisions logged, flagship still serving — and compare its would-have-served choices against reality for a week. Flip it on for the task families where the eval says the floor is solid, with shadow sampling and the escalation-rate alert live from day one. Then let six weeks of cascade logs decide whether a trained router is worth building, and revisit the ladder every time a vendor reprices — which, on current form, is every quarter.

None of this is exotic engineering. The cascade is twenty lines, the eval is an afternoon, the monitoring is four queries. What separates teams that cut spend by half without incident from teams that either overpay forever or rout quality by accident is not the router — it is the measurement discipline around it. Build the instrument before the optimisation, and the optimisation stops being a gamble.