Mixed-Vendor GPU Fleets: Inference on NVIDIA, AMD and Cloud TPUs

What you need to know

A mixed-vendor fleet is an operational answer to a supply problem, not an architectural preference. Teams end up serving inference on more than one kind of accelerator because the capacity they wanted was not available where they needed it, or because the price difference between two vendors grew large enough to fund the engineering, or because a data-residency obligation pinned them to a region where only one accelerator type is offered.

Three things follow from that, and they shape everything below. First, portability is a property you build deliberately in advance, not something you discover you have when the pager goes off. Second, the only comparison that survives contact with reality is cost per million output tokens at a fixed latency target, measured on your own traffic — not dollars per GPU-hour, and certainly not peak FLOPs from a vendor slide. Third, the ops burden of a second backend is close to fixed regardless of how much traffic flows through it, which means there is a scale below which a single vendor is unambiguously the right answer.

If you take one action after reading this, make it the cheap one: put a stable, OpenAI-compatible internal contract in front of whatever you run today, so that the day you need a second accelerator, the change is a deployment decision rather than an application rewrite.

Why anyone runs a mixed fleet

Let us be honest about the reasons, because the stated reason and the real reason are usually different.

Capacity, first and mostly

The dominant reason is that the accelerator you wanted was not available in the region you needed. A team in Pune building for Indian users wants capacity in an AWS Mumbai region; a Cambridge research group with a UK customer base wants London. Neither wants to serve traffic from Virginia. When a specific instance family in a specific region has no quota available on the timeline you are shipping to, you have three options: wait, serve from further away and pay the latency, or take whatever accelerator that region can actually give you. Plenty of teams take the third option and then reverse-engineer a strategic narrative for it afterwards.

This is worth naming plainly because it changes the engineering brief. If capacity is the driver, you are not optimising for the best accelerator — you are optimising for the ability to accept whichever accelerator is available at short notice without a two-month migration.

Price arbitrage

The second reason is a genuine price gap. Between the large clouds, the specialist GPU neo-clouds and regional providers in both India and the UK, on-demand rates for broadly comparable serving nodes are not the same number, and committed-use pricing moves the spread again. We deliberately quote no figures here: published rates change, discounts are negotiated rather than listed, and the only ones that should drive a decision are the ones on your own contract on the day you decide. The catch, covered below, is that the gap on a pricing page is not the gap you get, because achieved throughput differs too.

Negotiating leverage

The third reason is commercial rather than technical, but real. A team that can credibly move a workload has a different conversation at contract renewal. You do not need to move most of your traffic to get the benefit; a warm second backend serving your batch jobs is itself the leverage.

Data residency

The fourth reason is regulatory and it is the least negotiable. Neither the DPDP Act nor the UK GDPR is a blanket localisation mandate — both regulate cross-border transfer rather than forbidding it outright — but plenty of workloads end up pinned to a region anyway, by a sectoral rule, by a government notification restricting transfers to particular countries, or, most often, by a customer contract that simply says the data stays here. Once a workload is pinned that way, your accelerator choice collapses to whatever the compliant regions actually offer — sometimes a narrower menu than the global one. Routing by jurisdiction before routing by hardware is a design constraint worth handling explicitly; the mechanics are in our guide to data residency for AI apps under DPDP and GDPR.

The portability layer

Portability is not a single switch. It is four separate decisions, each of which you can get wrong independently.

Open-weight models are the precondition

If your workload runs on a proprietary hosted model, you have vendor choice at the API level but no accelerator choice at all, because you never touch the hardware. Everything here assumes you are serving open-weight models on infrastructure you control or rent. Open weights are the precondition; without them there is nothing to move.

A runtime with more than one backend

The single highest-leverage choice is serving through a runtime that already targets multiple accelerator families, so that the porting work has been done upstream by people who do it full time. As of August 2026, vLLM documents support for both NVIDIA CUDA and AMD ROCm backends, and AMD maintains its own ROCm documentation covering the driver and library stack underneath. For Google TPUs the equivalent path is the XLA compiler ecosystem: JAX compiles through OpenXLA to TPU targets, and Google publishes its own Cloud TPU documentation for the hardware and runtime. Alternative runtimes exist and are worth knowing about — SGLang and NVIDIA's TensorRT-LLM among them — with different trade-offs between raw performance on a single target and breadth of hardware support.

The practical rule: prefer a runtime whose multi-backend support is a maintained, tested part of the project rather than a community fork you would be maintaining yourself. If you are starting from scratch, our walkthrough of self-hosting an open-weight LLM with vLLM in production is the right foundation to build the second backend on top of.

No hand-written kernels in your own code

The fastest way to make a workload unportable is to write your own CUDA. A custom attention variant, a fused sampling kernel, a bespoke tokeniser accelerated by hand — each is a permanent tax you will pay again for every new accelerator family. There are workloads where a hand-written kernel is genuinely the difference between viable and not, and if you are in one of them you already know. For everyone else, take the runtime's kernel and spend the engineering time elsewhere.

An OpenAI-compatible HTTP surface as the internal contract

The contract between your application and your fleet should be an HTTP API with a stable schema, and in practice that means the OpenAI-compatible chat-completions surface, because every mainstream runtime speaks it. Callers send a model name and a request; they never learn which silicon answered. That indirection is what turns a hardware migration into a routing-table edit.

This is also where a gateway earns its place. A gateway gives you one point at which to attach routing policy, retries, per-backend rate limits and cost accounting, without every service reimplementing them — the landscape is covered in our comparison of LLM gateways including LiteLLM, OpenRouter and Portkey.

Comparing the economics properly

The most common mistake in mixed-fleet planning is comparing the wrong unit. Dollars per GPU-hour tells you nothing about how much work the device does. Peak FLOPs tells you even less, because large language model serving is frequently bound by memory bandwidth and by how efficiently the runtime batches requests, not by arithmetic throughput. A device with impressive headline FLOPs and insufficient memory to hold your model plus its key-value cache at your target concurrency is a bad buy at any price.

The unit that matters is cost per million output tokens at a fixed latency service level objective. Fixing the SLO is the part people skip, and it is the part that makes the comparison honest: any accelerator looks cheap if you allow unlimited queueing, because you can push batch size until throughput saturates and latency becomes unacceptable.

cost_per_1M_output_tokens
    = (node_hourly_rate / achieved_output_tokens_per_second) * 1e6 / 3600

where:
  node_hourly_rate            = total hourly cost of the whole serving node
                                (all accelerators + host + storage + egress share)
  achieved_output_tokens_per_second
                              = sustained aggregate output throughput measured
                                at the highest concurrency that STILL MEETS:
                                  TTFT p95 <= your target
                                  TPOT p95 <= your target
                                  error rate <= your target

Two refinements matter in practice. Use the cost of the whole node, not of one accelerator, because you pay for the host, the local storage and the idle sibling devices whether you use them or not. And use sustained throughput measured over a realistic window, not a burst, because thermal behaviour and memory fragmentation both show up over minutes rather than seconds.

The table below is a framework, not a benchmark. Every number in it is an illustrative placeholder chosen to demonstrate the arithmetic, and none of it should be read as a measurement of any vendor's hardware.

Comparison framework for accelerator classes. All figures are illustrative placeholders for demonstrating the calculation only — they are not vendor benchmarks, not quotations, and not measurements. Rates vary by region, contract, commitment term and provider, and throughput varies by model, quantisation and request shape. Re-measure every cell on your own workload before making a decision.

Accelerator class	Memory per device (illustrative)	Node rate per hour (illustrative)	Achieved output tok/s at your SLO (illustrative)	Derived cost per 1M output tokens
Vendor A, high-memory class	[fill in]	$X per hour	T tok/s	(X / T) × 277.8
Vendor B, high-memory class	[fill in]	$Y per hour	U tok/s	(Y / U) × 277.8
Cloud TPU class	[fill in]	$Z per hour	V tok/s	(Z / V) × 277.8
Older/commodity GPU class	[fill in]	$W per hour	S tok/s	(W / S) × 277.8
Interruptible / spot of any class	as above	discounted rate	as above, minus restart overhead	add an interruption-tax term

The constant 277.8 is simply 1e6 / 3600. Fill the table in yourself with rates pulled from your actual provider contracts on the day you make the decision, and throughput measured with the method in the next section. Then re-run it at renewal, because both columns move.

If you serve interruptible capacity, add an interruption tax covering re-warm time, lost in-flight work and the standby capacity you keep to absorb a reclaim; the mechanics are in our spot and pre-emptible GPU checkpoint-and-resume playbook. For the full picture including retries and failures, escalate from cost per million tokens to cost per successful task using the method in LLM unit economics and cost per task.

Routing by workload shape, not by preference

Once you have more than one pool, the temptation is to route by whichever pool you like best, or by whichever is cheapest per hour, and to let a load balancer spread traffic evenly. Both are wrong. Route by the shape of the request, because different accelerator classes suit different shapes and the mismatch is expensive in both directions.

The dominant shape variable is memory. Long-context requests and large mixture-of-experts models need devices with enough high-bandwidth memory to hold the weights plus a key-value cache that grows linearly with context length and concurrency. Put those on your highest-memory tier and nowhere else; the capacity arithmetic for the largest models is covered in serving a trillion-parameter MoE on your own hardware. Short, bursty chat traffic with modest context is the opposite: it wants whatever is cheap and available, and it benefits far more from having enough replicas to absorb a spike than from having the fastest device per replica. Batch and offline work — nightly summarisation, embedding backfills, evaluation sweeps, document extraction pipelines — should land on whatever is cheapest and most interruptible, because it can tolerate restarts and has no user waiting.

Mapping workload shape to fleet tier. Adapt the tier names to your own pools; the mapping logic is the transferable part.

Workload shape	Dominant constraint	Fleet tier	Failure behaviour
Long-context RAG, 100k+ token prompts	KV cache memory	High-memory tier only	Queue rather than spill to a smaller device
Large MoE model, interactive	Weight residency and interconnect	High-memory tier only	Reject with a clear error; do not silently downgrade the model
Short interactive chat, <8k context	Replica count and TTFT	Commodity tier, any vendor	Fail over across vendors freely
Streaming assistant, latency-sensitive	TPOT p95	Whichever tier meets TPOT at lowest cost	Fail over within the same latency class
Nightly batch, embeddings, backfills	Cost per token only	Cheapest interruptible tier	Checkpoint and resume
Evaluation sweeps and regression runs	Reproducibility per backend	Pinned to each backend in turn	Never fail over — a failover invalidates the comparison
Residency-constrained traffic	Jurisdiction	In-region pools only	Fail closed, never cross the border

Note the last row. Residency routing must be evaluated before hardware routing and must fail closed. A fallback that quietly sends an Indian user's request to a London pool because Mumbai was busy is a compliance incident, not a resilience feature.

Here is the shape of a router that selects a pool by model, context length and latency class, with health checks and an explicit residency gate.

from dataclasses import dataclass, field

@dataclass
class Pool:
    name: str
    backend: str              # "cuda" | "rocm" | "tpu"
    base_url: str             # OpenAI-compatible endpoint
    region: str               # "ap-south-1", "eu-west-2", ...
    models: set               # model ids actually loaded here
    max_context: int          # hard ceiling for this tier
    latency_class: str        # "interactive" | "standard" | "batch"
    cost_per_1m_tokens: float # measured, not quoted
    healthy: bool = True

    def accepts(self, req) -> bool:
        return (
            self.healthy
            and req.model in self.models
            and req.context_tokens <= self.max_context
            and LATENCY_RANK[self.latency_class] <= LATENCY_RANK[req.latency_class]
        )

LATENCY_RANK = {"interactive": 0, "standard": 1, "batch": 2}

@dataclass
class Request:
    model: str
    context_tokens: int
    latency_class: str = "standard"
    residency: str | None = None      # "IN" | "UK" | None
    tried: set = field(default_factory=set)

REGION_JURISDICTION = {
    "ap-south-1": "IN",
    "eu-west-2": "UK",
}

class NoCapacity(Exception):
    pass

def select_pool(req: Request, pools: list[Pool]) -> Pool:
    """Pick the cheapest healthy pool that can serve this request shape."""
    candidates = [p for p in pools if p.name not in req.tried and p.accepts(req)]

    # Residency gate runs BEFORE any hardware preference, and fails closed.
    if req.residency is not None:
        candidates = [
            p for p in candidates
            if REGION_JURISDICTION.get(p.region) == req.residency
        ]

    if not candidates:
        raise NoCapacity(f"no pool for {req.model} @ {req.context_tokens} tokens")

    # Cheapest first. Ties broken by the pool with the most headroom.
    candidates.sort(key=lambda p: (p.cost_per_1m_tokens, -p.max_context))
    return candidates[0]

def call_with_fallback(req: Request, pools: list[Pool], send, attempts: int = 3):
    """Try pools in cost order; a hardware failure moves to the next backend."""
    last_error = None
    for _ in range(attempts):
        pool = select_pool(req, pools)
        try:
            return send(pool, req), pool
        except (ConnectionError, TimeoutError) as exc:
            last_error = exc
            pool.healthy = False       # health checker will re-enable it
            req.tried.add(pool.name)
    raise NoCapacity(f"all backends exhausted: {last_error}")

Two details there are load-bearing. The residency filter runs before the cost sort, so cheapness can never override jurisdiction. And cost_per_1m_tokens is a measured field, refreshed from your own benchmarking, not a number typed in from a pricing page.

Benchmarking your own fleet

You cannot use vendor benchmarks for this decision, and not because vendors are dishonest. A published benchmark fixes a model, a quantisation, a batch size and a request distribution that are almost certainly not yours. Your prompt and completion length distributions, your concurrency profile across the day and your tolerance for latency variance determine achieved throughput, and none of them appear in anyone else's numbers.

The method is not complicated, it is just disciplined.

1. Fix the model. Same weights, same revision, on every backend. If you must use different quantisation formats per backend because the formats do not transfer, record that as a variable and run your evaluation suite on each.
2. Capture a real trace. Export a representative window of production requests — prompt token counts, completion token counts, arrival timestamps. A week is usually enough to include your peaks. Strip content if you must; the shape is what matters.
3. Replay, do not synthesise. Synthetic fixed-length prompts flatter batching and will overstate throughput. Replay your own distribution.
4. Sweep concurrency. Step through concurrency levels until the SLO breaks. The interesting number is the highest concurrency that still passes, not the maximum throughput at any latency.
5. Record four metrics at each point. Time to first token at p95, time per output token at p95, aggregate output tokens per second, and error rate. Report all four; a throughput figure without its latency percentiles is not a result.
6. Repeat on a different day. Shared infrastructure varies. A single run is an anecdote.

"""Replay a production trace against an OpenAI-compatible endpoint
and report TTFT p95, TPOT p95, throughput and error rate."""

import argparse, asyncio, json, statistics, time
import httpx

async def one_request(client, base_url, model, prompt_tokens, max_tokens, results):
    prompt = "word " * prompt_tokens          # shape-preserving filler
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": True,
        "temperature": 0.0,
    }
    started = time.perf_counter()
    first_token_at = None
    n_out = 0
    try:
        async with client.stream("POST", f"{base_url}/v1/chat/completions",
                                 json=payload, timeout=300) as resp:
            resp.raise_for_status()
            async for line in resp.aiter_lines():
                if not line.startswith("data: "):
                    continue
                body = line[6:]
                if body.strip() == "[DONE]":
                    break
                chunk = json.loads(body)
                delta = chunk["choices"][0]["delta"].get("content")
                if not delta:
                    continue
                if first_token_at is None:
                    first_token_at = time.perf_counter()
                n_out += 1
    except Exception as exc:
        results.append({"ok": False, "error": type(exc).__name__})
        return

    finished = time.perf_counter()
    if first_token_at is None or n_out < 2:
        results.append({"ok": False, "error": "no_output"})
        return

    results.append({
        "ok": True,
        "ttft": first_token_at - started,
        "tpot": (finished - first_token_at) / (n_out - 1),
        "out_tokens": n_out,
        "wall": finished - started,
    })

async def run_level(trace, base_url, model, concurrency):
    sem = asyncio.Semaphore(concurrency)
    results = []
    limits = httpx.Limits(max_connections=concurrency * 2)
    async with httpx.AsyncClient(limits=limits) as client:
        async def guarded(row):
            async with sem:
                await one_request(client, base_url, model,
                                  row["prompt_tokens"], row["completion_tokens"],
                                  results)
        t0 = time.perf_counter()
        await asyncio.gather(*(guarded(r) for r in trace))
        elapsed = time.perf_counter() - t0
    return results, elapsed

def p95(values):
    if not values:
        return float("nan")
    ordered = sorted(values)
    return ordered[min(len(ordered) - 1, int(round(0.95 * (len(ordered) - 1))))]

def report(label, results, elapsed):
    ok = [r for r in results if r["ok"]]
    total_out = sum(r["out_tokens"] for r in ok)
    print(json.dumps({
        "level": label,
        "requests": len(results),
        "error_rate": round(1 - len(ok) / max(len(results), 1), 4),
        "ttft_p95_s": round(p95([r["ttft"] for r in ok]), 3),
        "tpot_p95_s": round(p95([r["tpot"] for r in ok]), 4),
        "output_tokens_per_s": round(total_out / elapsed, 1),
        "wall_s": round(elapsed, 1),
    }))

async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--trace", required=True,
                    help="JSONL with prompt_tokens and completion_tokens per line")
    ap.add_argument("--base-url", required=True)
    ap.add_argument("--model", required=True)
    ap.add_argument("--levels", default="1,4,8,16,32,64,128")
    args = ap.parse_args()

    with open(args.trace) as fh:
        trace = [json.loads(line) for line in fh if line.strip()]

    for level in (int(x) for x in args.levels.split(",")):
        results, elapsed = await run_level(trace, args.base_url, args.model, level)
        report(level, results, elapsed)
        await asyncio.sleep(20)   # let the server settle between levels

if __name__ == "__main__":
    asyncio.run(main())

Run that against each backend in turn with the same trace file, take the highest concurrency level where TTFT p95, TPOT p95 and error rate all pass, and feed the corresponding output_tokens_per_s into the cost formula from the previous section. That is your comparison. Every other number you have seen was measured on somebody else's workload.

The ops burden nobody budgets for

Here is the part that does not appear in the business case. A second accelerator vendor roughly doubles several categories of work that were previously invisible because they only existed once.

Recurring costs of a second backend. These are approximately fixed — they scale with the number of backends, not with traffic volume.

Burden	What it actually means	Who pays it
Driver and runtime matrix	Kernel version, driver version, runtime version and framework version must be jointly compatible — per vendor, and re-verified on every upgrade	Platform team, continuously
Per-vendor container images	Two image lineages to build, scan, sign and keep patched; two sets of base-image CVE triage	Build and security, every release
Quantisation artefacts	Formats and kernels differ per backend; a quantised checkpoint that runs well on one may not load, or may not be fast, on another	ML engineering, every model update
Divergent numerics	Different kernels and accumulation orders shift outputs slightly; near a decision boundary a token choice can flip	Evaluation, per backend, per release
Observability parity	Device metrics, memory accounting and failure signatures differ; dashboards and alert thresholds are not reusable as-is	SRE, once per backend plus drift
On-call mental models	Responders need to recognise two sets of failure modes at 3am, in a rotation that may only see one of them monthly	Everyone on the rota, permanently
Capacity forecasting	Two quota conversations, two commitment structures, two renewal cycles	Whoever owns the budget

The divergent-numerics row surprises people. Outputs are not worse on the second backend; they are simply not identical. If anything downstream depends on exact strings — a cache keyed on the completion, a regex parser, a tuned threshold, a golden-file test — it can behave differently. The response is not to chase bit-exactness, which you will not get, but to treat every backend as a separate release target and run your full evaluation suite against each, every time.

The threshold below which one vendor wins

Say it plainly: if your annual accelerator spend is below roughly a hundred thousand pounds, or roughly a crore of rupees, and you have no residency obligation forcing your hand, a single vendor is the right answer. That figure is a rule of thumb, not a measured threshold — it is the order of magnitude at which, in our experience, a realistic price gap starts to fund a second platform engineer rather than borrow one. Derive your own version by pricing the burdens in that table in engineer-weeks and asking whether your plausible arbitrage covers them. The burdens are close to fixed. At small scale they consume a larger share of your engineering capacity than the price arbitrage returns, and the opportunity cost is measured in features you did not ship.

Two exceptions override the threshold. If you genuinely cannot get capacity in the region you need — and a Mumbai or London quota shortfall is a real constraint, not a hypothetical one — then availability trumps economics at any scale. And if a regulator or a customer contract pins you to a jurisdiction where only one accelerator type is offered, you have no decision to make. Outside those two cases, stay on one vendor and spend the saved effort on the abstraction layer, so that the option remains open.

A staged adoption path

When you do decide to go mixed, stage it. Each stage has a verification gate, and you should not pass the gate on optimism.

Stage 1: Abstraction first, still single-vendor

Put the OpenAI-compatible contract in front of your existing fleet and make every caller go through it. Remove direct references to instance types, device names and vendor libraries from application code. Introduce a model-name to pool mapping that lives in configuration rather than in code.

Verify: no service in your estate can name the hardware it is talking to; you can change which pool serves a model with a configuration change and no redeploy of callers; your cost accounting already reports cost per million tokens per pool.

Stage 2: Shadow the second backend

Stand up the second accelerator with the same model and mirror a copy of live traffic to it. It answers nobody. You are measuring, not serving. Run the benchmark harness against it, run your full evaluation suite against it, and diff the outputs against the primary to understand the numerical divergence before it can hurt you.

Verify: the model loads with an acceptable quantisation; the benchmark passes your SLO at a useful concurrency; the evaluation suite passes on this backend specifically; you have catalogued the output differences and confirmed nothing downstream depends on exactness; dashboards and alerts exist for the new device class.

Stage 3: Route batch and offline traffic

Move nightly jobs, embedding backfills and evaluation sweeps onto the second backend. This is the right first real workload because it tolerates latency variance, tolerates restarts and has no user waiting. It also runs enough volume to surface the slow-burning problems — memory leaks, fragmentation, thermal behaviour, driver instability under sustained load — that a shadow deployment will not.

Verify: a full week of batch load with no manual intervention; checkpoint-and-resume works under a real interruption; per-pool cost accounting shows the arbitrage you predicted; the on-call rota has handled at least one incident on the new backend unaided.

Stage 4: Route live traffic

Start with a small percentage of the least latency-sensitive interactive class, on one model. Hold it for a week. Increase in steps, watching TTFT p95 and TPOT p95 per pool rather than in aggregate, because an aggregate percentile will hide a bad pool behind a good one. Keep an instant drain path that moves all traffic back to the primary with a configuration change.

Verify: per-pool latency percentiles, not just aggregate; error rates and retry rates split by backend; user-facing quality metrics unchanged; the drain path rehearsed and timed, with the real number written down.

Stage gates at a glance.

Stage	Traffic on new backend	Primary gate to pass	Rollback
1. Abstraction	None	No caller knows the hardware	Not applicable
2. Shadow	Mirrored, unserved	Evaluations pass per backend; divergence catalogued	Delete the pool
3. Batch	Offline jobs only	One week unattended; cost arbitrage confirmed	Re-point the job queue
4. Live	Ramped percentage	Per-pool percentiles hold; quality unchanged	Timed, rehearsed drain

Conclusion: what to do next

A mixed-vendor fleet is a capability, and like most capabilities it is cheapest to acquire before you need it and most expensive to acquire during an incident. The asymmetry is the whole argument: building the abstraction layer costs a fortnight whether or not you ever use it, while retrofitting portability under capacity pressure costs a quarter and a great deal of goodwill.

So the practical sequence is: build the portability layer now regardless of your scale; measure your own fleet properly so you have a real cost per million tokens rather than a price-list number; and only add a second vendor when availability, residency or a genuinely large measured price gap justifies the fixed operational burden. If you are below the threshold and unconstrained, staying on one vendor is not a failure of ambition — it is the correct decision, and the abstraction layer keeps the option open for the day it changes.

If you take on the work, write it down. A documented migration with a real benchmark method, per-backend evaluation results and a rehearsed drain path is one of the most credible artefacts an infrastructure engineer can show, in Bengaluru or in Bristol, because very few people have actually done it end to end.