The AI Cost Engineer: Proving You Can Cut a Token Bill

What the role actually is, and the four titles it hides behind

There is a job that a large number of organisations in India and the UK are hiring for and almost none of them are advertising under its real name. The person in it owns the relationship between what an AI feature does and what it costs to run. They can state, to a meaningful number of decimal places, what one request costs. They can tell you which of your six AI features is consuming two-thirds of the inference bill, and why. And they can produce a plan that takes a third out of the total without the product team noticing anything except a faster first token.

As of September 2026, that work appears on job boards under at least four titles, and recognising them is the first practical skill here.

FinOps engineer is the most literal. It usually sits in a platform or cloud-economics function, reports somewhere near finance, and before 2024 meant reserved instances and untagged storage buckets. The AI mandate was bolted on during 2024 and 2025 and the people in the seat are frequently the ones who asked for it.

AI platform engineer is the same work described from the engineering side. The listing talks about gateways, routing, observability and provider abstraction; the cost ownership is implied by the fact that the gateway is where every token passes through, and whoever owns the gateway inevitably owns the meter.

ML infrastructure engineer skews towards teams that run their own accelerators. Here the currency is GPU-hours and utilisation rather than tokens, and the question is whether a cluster that is busy is actually productive. The arithmetic differs; the discipline does not.

Senior engineer who owns the bill has no title at all. It is the most common version of this role in practice: a respected engineer on a product team who, at some point during a budget conversation, became the person everyone looks at when the invoice arrives. In Indian GCCs and product companies in Bengaluru and Hyderabad this is very often how the function starts — an engineer builds a dashboard nobody asked for, the dashboard answers a question the CFO had been asking for a quarter, and six months later it is in their objectives. In UK scale-ups and financial-services teams the same transition tends to run through a formal cost-control exercise instead, because the finance function was already asking for unit economics on every line of the technology budget before generative AI arrived.

The titles differ. The competence is identical, and it is unusually easy to prove.

Why the demand appeared: 98 per cent tracking, 73 per cent overrunning

The clearest picture of this shift comes from the FinOps Foundation's State of FinOps 2026 report, released in February 2026. It draws on 1,192 respondents representing organisations with more than $83 billion in annual cloud spend, which makes it one of the few datasets in this area large enough to argue from.

Three findings matter for anyone thinking about this as a career.

First, the adoption curve is effectively complete. Ninety-eight per cent of respondents now manage AI spend, up from 31 per cent two years earlier. That is not growth, it is a phase change, and it happened faster than any comparable shift in the same survey. For context within the same dataset, 90 per cent manage SaaS spend, 64 per cent manage licensing and 48 per cent manage data centre costs. AI spend went from a minority concern to the most universally managed category on that list in under twenty-four months.

Second, the priority ordering has moved. AI cost management is the top forward-looking priority among the practitioners surveyed, and AI value management is the single most in-demand skill teams say they are trying to add. That second phrase is the one to read carefully, because it is a statement about hiring rather than about tooling. Organisations are not saying they need a better dashboard; they are saying they need people who can reason about whether the spend is producing anything.

Third, and most usefully for you: 73 per cent of AI projects still blow their budget despite the tracking.

The reason the gap exists is that most cost visibility is aggregate. A monthly provider invoice tells you inference spend rose 40 per cent. It does not tell you that a retrieval change three weeks ago increased the average context window from 4,000 to 9,000 tokens on one feature that happens to carry 60 per cent of your request volume. The survey identifies exactly this: the single most-requested capability across the entire respondent base is granular monitoring of AI spend — tokens, LLM requests and GPU utilisation. Not more reporting. More resolution.

The same pressure is reshaping how individual engineers experience the budget. Companies are increasingly issuing developers personal token budgets as a managed resource, in the same way cloud credits were allocated a decade ago. That is a meaningful cultural signal: it means cost has stopped being a platform-team abstraction and has become something an individual contributor is expected to have an opinion about. Everyone is now adjacent to this work. Very few people are good at it.

The five things a cost engineer is actually expected to do

Strip away the titles and the role reduces to five capabilities. A candidate who can demonstrate three of them is hireable. A candidate who can demonstrate all five is, as of September 2026, difficult to hire against.

Notice what is not in that table. Nothing about negotiating with vendors, nothing about picking the fashionable model, nothing about writing a policy document. This is an engineering role in which the output happens to be denominated in currency. If you want the finance-facing complement to it — the version of this argument you make to a budget holder rather than to a platform team — the mechanics of building an internal business case for an AI project is a different skill worth having alongside, but it is not what gets tested in the technical loop.

Where the three caching layers actually differ

Capability three deserves its own treatment, because it is the one most often described in a single undifferentiated word. Teams say "we cache" and mean any of three unrelated mechanisms with different savings profiles, different failure modes and different operational costs. Conflating them is the most common technical tell in an interview.

The mechanical detail that separates the three is worth internalising, because interviewers probe it. Exact-match and semantic caching both bypass the model: on a hit, no inference happens at all, which is why they are the only layers that reduce output cost. Prefix caching does not bypass the model; it lets the provider skip recomputing the attention key-value tensors for a prefix it has seen before. That saves the input-side charge for those tokens and shortens time-to-first-token, but every output token is generated fresh and billed at the full output rate.

The rubric: answering "how would you cut our LLM bill by 40 per cent?"

This question, or a close variant of it, appears in almost every interview loop for this work. It is not a knowledge question. It is a diagnostic for whether the candidate reaches for a cause or a remedy first, and the difference between a strong and a weak answer shows up in the opening sentence.

Four things separate those answers, and an interviewer is usually scoring all four without saying so.

It asks for the cost breakdown first. This is the whole rubric compressed into one behaviour. A candidate who optimises before measuring will optimise the wrong thing at a real job too, and everyone in the room knows it.

It separates input from output spend. As the worked example below shows, these two halves respond to entirely different interventions, and a candidate who treats "tokens" as one quantity will propose caching for an agent workload where caching is nearly irrelevant.

It states a prior and then offers to test it. "My prior is that spend is concentrated" is a falsifiable claim about the system, which is a different register from a confident recommendation. Senior engineers reason in priors; junior ones reason in best practices.

It attaches risk to each proposed saving. Every cost lever trades something away — quality, latency, freshness, or operational complexity. A ranked list where each row carries a projected saving and its cost is a plan. A list of techniques is a search result.

If you want to rehearse the underlying reasoning before you meet it in a room, the discipline of forecasting an LLM bill with a cost model before you launch is the same mental motion run forwards instead of backwards: you are predicting a number from a token budget rather than explaining one after the invoice arrives. Teams that can do the first rarely need the second.

The portfolio artefact: publish a cost teardown

Interview answers are cheap. The thing that ends the conversation in your favour is a published teardown, because it is the rare artefact that cannot be produced by someone who has only read about the subject. Here is the full specification.

Pick a real open-source AI application. Not a toy, not your own demo. Something with a repository, a real prompt structure and a retrieval or tool layer — a documentation assistant, an open-source support agent, a code-review bot. The point of using someone else's system is that you cannot quietly redesign it to make your numbers better.

Instrument it. Wrap every model call so that you log, per request: the feature or code path, input tokens, cached input tokens, output tokens, model, latency, and a request identifier. This is unglamorous and it is where the credibility comes from, because the log is what makes the rest arithmetic rather than assertion.

Build the baseline cost model. Compute cost per request with input, cached input and output separated. Do not collapse them.

from dataclasses import dataclass

# Rate card in USD per 1,000,000 tokens.
# Worked-example assumption, not a quote - substitute your provider's
# published rates and re-date them whenever you refresh the teardown.
RATES = {
    "input":        0.30,
    "cached_input": 0.03,   # 90% read discount on a cached prefix
    "output":       1.20,
}


@dataclass
class Usage:
    request_id: str
    feature: str
    input_tokens: int          # billed at the full input rate
    cached_input_tokens: int   # prefix served from the provider's KV cache
    output_tokens: int


def request_cost_usd(u: Usage) -> dict:
    """Cost of one request, with the three components kept separate."""
    billed_input = u.input_tokens * RATES["input"] / 1_000_000
    cached_input = u.cached_input_tokens * RATES["cached_input"] / 1_000_000
    output = u.output_tokens * RATES["output"] / 1_000_000
    return {
        "input_usd": billed_input,
        "cached_input_usd": cached_input,
        "output_usd": output,
        "total_usd": billed_input + cached_input + output,
    }


def unit_economics(log):
    """Aggregate a usage log into per-feature unit economics."""
    by_feature = {}
    for u in log:
        c = request_cost_usd(u)
        agg = by_feature.setdefault(u.feature, {
            "requests": 0,
            "input_usd": 0.0,
            "cached_input_usd": 0.0,
            "output_usd": 0.0,
            "total_usd": 0.0,
        })
        agg["requests"] += 1
        for k in ("input_usd", "cached_input_usd", "output_usd", "total_usd"):
            agg[k] += c[k]

    for agg in by_feature.values():
        agg["cost_per_request_usd"] = agg["total_usd"] / agg["requests"]
        # The single most useful derived number in the whole model:
        # if this is high, caching will not save you.
        agg["output_share"] = agg["output_usd"] / agg["total_usd"]
    return by_feature

That last derived field, output_share, is the one to lead with in your write-up. It decides which half of the playbook applies before you spend a day on the wrong one.

Instrument the cache layers separately. This is the part almost nobody does, and therefore the part that distinguishes the artefact.

import hashlib
from collections import Counter


class CacheTelemetry:
    """Per-layer cache accounting. One aggregate hit rate is not actionable."""

    LAYERS = ("exact", "semantic", "prefix")

    def __init__(self):
        self.lookups = Counter()
        self.hits = Counter()
        self.tokens_avoided = Counter()

    def record(self, layer, hit, tokens_avoided=0):
        if layer not in self.LAYERS:
            raise ValueError(f"unknown cache layer: {layer}")
        self.lookups[layer] += 1
        if hit:
            self.hits[layer] += 1
            self.tokens_avoided[layer] += tokens_avoided

    def hit_rate(self, layer):
        n = self.lookups[layer]
        return (self.hits[layer] / n) if n else 0.0

    def report(self):
        return {
            layer: {
                "lookups": self.lookups[layer],
                "hit_rate": round(self.hit_rate(layer), 4),
                "tokens_avoided": self.tokens_avoided[layer],
            }
            for layer in self.LAYERS
        }


def normalized_key(prompt: str) -> str:
    # Collapse whitespace and case before hashing, otherwise trivial
    # formatting differences destroy an otherwise healthy hit rate.
    normalized = " ".join(prompt.split()).lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()


def answer(prompt, telemetry, exact_store, semantic_index, model):
    """Three layers, each measured on the way past."""
    key = normalized_key(prompt)

    cached = exact_store.get(key)
    telemetry.record("exact", cached is not None,
                     tokens_avoided=(len(prompt) // 4) if cached else 0)
    if cached is not None:
        return cached

    match = semantic_index.nearest(prompt, threshold=0.93)
    telemetry.record("semantic", match is not None,
                     tokens_avoided=match.tokens_avoided if match else 0)
    if match is not None:
        exact_store.set(key, match.response)
        return match.response

    response = model.complete(prompt)
    # Prefix caching is reported by the provider in the usage payload.
    telemetry.record("prefix", response.usage.cached_input_tokens > 0,
                     tokens_avoided=response.usage.cached_input_tokens)
    exact_store.set(key, response.text)
    semantic_index.add(prompt, response.text)
    return response.text

The worked cost model, with arithmetic you can check

Pricing as of September 2026. The rate card used below is a worked assumption, stated so the arithmetic is checkable, not a quote from any provider: $0.30 per 1M input tokens, $0.03 per 1M cached input tokens (a 90 per cent read discount), $1.20 per 1M output tokens. Substitute your own rates and re-run; the method survives a price change, the figures do not.

The system: a document-question-answering assistant with a 10,000-token stable prefix (a 4,000-token system prompt plus 6,000 tokens of fixed policy context), roughly 500 tokens of per-request variable input, and 700 output tokens per answer. Volume: 300,000 requests per month.

Check the arithmetic yourself, because a reader of your teardown will. The prefix carries 10,000 × 300,000 = 3,000M tokens a month. At an 80 per cent hit rate, 2,400M of those are served from cache at $0.03 per 1M, which is $72.00, and the remaining 600M are billed in full at $0.30 per 1M, which is $180.00. The prefix therefore costs $252.00 after caching against $900.00 before — a saving of 72 per cent on that prefix's input cost, which is the general rule of thumb for an 80 per cent hit rate against a 90 per cent read discount. Total spend falls from $1,197.00 to $549.00, a reduction of $648.00, or 54.1 per cent of the bill.

That relationship is not universal, and the honest version of a teardown says so. Run the same arithmetic for a customer support application with a 4,000-token system prompt and 6,000 tokens of RAG context, at a 95 per cent hit rate, and the total saving lands well above the model above — driven by the higher hit rate and by that system's particular input-to-output ratio. Work it through yourself rather than taking a headline percentage on trust; that is the entire point of publishing a breakdown. Two applications with identical prompt structures and different answer lengths will report wildly different headline savings, which is precisely why publishing the breakdown matters more than publishing the percentage.

Publish the methodology. The write-up is the artefact, not the repository. State the system, the instrumentation, the rate card and its date, the before and after tables, the per-layer hit rates, and one section headed with the limitation — what you could not cut and why. A teardown that reports only wins reads as a demo. The format that gets read is covered in writing up an AI project so it survives a ninety-second scan; the constraint that matters is that a hiring manager decides within two screens whether you measured or guessed.

The trap: optimising the wrong token

Here is the mistake that catches competent engineers, and the reason the rubric above insists on separating input from output before proposing anything.

Every caching layer that saves input cost saves nothing on output. Prefix caching, by construction, only avoids recomputing the prefix; the model still generates every output token and you are billed for every one of them at the output rate, which on most rate cards is three to four times the input rate. So the value of an input-side optimisation depends entirely on what fraction of your bill is input.

Take the same document-QA system and change one variable: make it agentic. Instead of a 700-token answer, a run now involves a planning step, four tool calls with reasoning between them, and a synthesis — call it 9,400 output tokens across the run. Everything else is identical: the same 10,000-token stable prefix, the same 500 tokens of variable input, the same 300,000 runs a month, the same caching work producing the same 80 per cent hit rate.

The input side fell by $648.00 in both cases — identical work, identical saving, a 68.6 per cent cut to input cost. In the first workload that is 54 per cent of the bill. In the second it is 15 per cent. Nothing about the optimisation changed; only the denominator did.

Two other denominators deserve the same scepticism. Language is one: if your traffic is substantially Hindi, Tamil, Bengali or any other Indic script, the tokeniser itself inflates every count before any optimisation applies, and the arithmetic of the tokeniser tax on Indic languages changes your baseline rather than your savings rate. Hardware is the other: the marginal cost of inference is not independent of the memory market, and the 2026 DRAM shortage raising the inference cost floor is the sort of exogenous move that resets every model in this article. If you are weighing running your own accelerators against paying per token, the crossover point is genuinely sensitive to that market, and the self-hosting break-even under memory price inflation is the calculation to run rather than a rule of thumb to quote.

Scheduling: the lever that requires no engineering at all

The fifth capability is the one candidates most often forget, and it is the cheapest to demonstrate. Some providers price the same model differently by time of day. As of September 2026, DeepSeek prices V4.1 Flash at $0.30 per 1M input and $1.20 per 1M output at peak, and $0.15 per 1M input and $0.60 per 1M output off-peak, with off-peak cache-hit input at $0.003 per 1M. That is a halving of the headline rates for moving work into a window, and it requires no change to a prompt, a model or an evaluation.

The practical catch is one that matters specifically for teams spanning India and the UK: a provider's off-peak window falls at different local hours depending on where your batch jobs are orchestrated and where your operations team sits. A window that lands comfortably in the small hours for a Bengaluru or Hyderabad team may land mid-afternoon for a London or Edinburgh team, or the reverse, and the difference decides whether the job runs unattended or needs someone awake. Teams with both markets frequently end up running the same nightly job twice against two different windows, and the person who worked that out in advance rather than after a failed overnight run is the person who reads the provider's published schedule in UTC and converts it to both IST and GMT or BST before writing the cron entry.

Classify every workload by latency tolerance first: a synchronous user-facing request cannot move, but nightly summarisation, index rebuilds, evaluation runs, backfills and bulk classification usually can. That classification is a half-day of work and is often the single largest saving available to a team that has never done it.

Getting the first one of these roles without the title

There are two routes and they are not mutually exclusive. The internal one is faster if you are already employed; the external one travels with you.

The internal route: volunteer to own the bill. As of September 2026, hardly anybody wants this. That is the opportunity. Ask for read access to the provider usage data and the gateway logs, and produce, unasked, a one-page breakdown of spend by feature with cost per request for the top three. Do not propose anything in that first document — just show the number nobody else has. In most organisations this is the entire distance between being a senior engineer and being the person the CFO's office emails about AI. Then pick the single largest attributable line and run one intervention end to end, with a measured before and after. One completed intervention with real numbers beats a roadmap of five proposed ones, and it gives you a result you can describe in an interview at a different company later.

The external route: the published teardown. Build the artefact specified above on an open-source application and publish it. It takes a weekend or two and it is portable, verifiable and rare. The reason it works disproportionately well is that this skill is otherwise almost impossible to evidence from outside a company — you cannot show a hiring manager your employer's invoice, but you can show them your arithmetic on a system they can clone and check.

What to put on a profile. Write in the language of results, not activity. "Cut inference spend 38 per cent on a 40,000-request-a-day retrieval feature by restructuring the prompt prefix and adding a semantic cache at a 0.93 threshold; cache hit rates 22 per cent exact, 19 per cent semantic, 81 per cent prefix" tells a reader everything. "Worked on LLM cost optimisation" tells them nothing and is indistinguishable from having attended a meeting about it. Include the per-layer hit rates specifically, because that single line is a competence signal that cannot be faked by someone who has not instrumented a system. And include the limitation — the thing you could not cut and why — since every experienced reader knows a ceiling exists and trusts the candidate who names theirs.

What this looks like in 18 months

Three things seem reasonably safe to project from where the discipline stands in September 2026, and it is worth being explicit that these are projections rather than measurements.

The first is that granular AI cost monitoring stops being a differentiator and becomes table stakes. It is the most-requested capability in the State of FinOps 2026 data, which means every observability vendor and every model gateway is building it. When per-token, per-feature attribution ships by default, the scarce skill is no longer producing the number — it is deciding what to do about it, which pushes the role further towards judgement and away from instrumentation. Build the instrumentation skill now because it is how you learn the judgement, not because the instrumentation itself will stay scarce.

The second is that the unit of measurement moves from cost to value. The survey language is already there: the most in-demand skill is described as AI value management, not AI cost management. The question shifts from "what does this feature cost" to "what is the gross margin on this feature, and should it exist". That is a materially harder question, it requires a working relationship with whoever owns the product's revenue line, and it is where this role starts touching decisions rather than reporting on them.

The third is that developer-level token budgets normalise. Companies are already issuing them as a managed resource. Within eighteen months it seems likely that an engineer in a Bengaluru product team or a London scale-up will think about their token allocation roughly the way they thought about a cloud environment in 2026 — as a constraint they are individually accountable for. When that happens, cost literacy stops being a specialism and becomes a baseline expectation, and the people who moved early will be the ones setting the budgets rather than spending against them.

None of which requires waiting. The gap between 98 per cent of organisations watching their AI spend and 73 per cent of AI projects still overrunning is open as of September 2026, it is measurable, and closing even a small part of it is the most legible piece of work an engineer can do in this market. Instrument something. Publish the arithmetic. The role finds you after that.