Break Into Inference Engineering: The SWE-Serve Skill Map

Why this specialisation suddenly became legible

For most of the past three years, "inference engineering" was a phrase people used with confidence and defined differently every time. One team meant CUDA kernels. Another meant Kubernetes and autoscaling. A third meant quantisation. A candidate could not prepare for the role, because nobody could describe its boundaries, and a hiring manager could not screen for it, because there was no shared vocabulary and no artefact that settled the question. The result was a market where the work was clearly valuable and the skill was almost impossible to demonstrate.

That changed in September 2026 with the publication of SWE-Serve, a benchmark described in arXiv preprint 2609.26777 by Jennifer Williams, Dave Farris, Jeff Farris and Jiantao Jiao, under the title SWE-Serve: Benchmarking Agentic Engineering for Production Inference Serving. The reason it matters for your career is not that it measures agents. It is that, in order to measure anything, the authors had to write down what production inference engineering consists of — and they did it by taking 53 repository-grounded tasks derived from merged pull requests in SGLang, a real, widely deployed serving engine. Those are not invented exercises. They are changes that real maintainers reviewed and shipped.

One finding in particular functions as a job description. The authors report that end-to-end serving tests reject roughly one third of the patches that pass their other evaluations. Read that slowly. A third of changes that look correct, and that satisfy conventional checks, fail once they have to survive persistent state, concurrent requests and integration with the running server. The skill being measured is not writing the patch. It is making the patch survive contact with a server that is holding state and serving many callers at once.

We covered the benchmark itself, and what it says about agents doing this work, in our news piece on SWE-Serve. This guide is the other half of that story: what a human should do with it.

What inference engineering actually covers

The most useful thing SWE-Serve gives a newcomer is a documented span of the work. The 53 tasks are split into 41 GPU tasks, run on H100 hardware, and 12 CPU tasks. They span six inference-engineering families. The abstract does not enumerate all six names, so do not let anyone hand you a tidy list of six that was reverse-engineered from a blog post; what the paper does document is the territory the tasks cover, which includes model support, runtime execution and public APIs. The tasks test correctness including integration, through end-to-end execution, persistent state and concurrent requests. The reference implementation runs with Harbor, and the code is published at github.com/NVIDIA/swe-serve.

Two observations from that split change how you should plan.

The first is that roughly a quarter of the tasks do not need a GPU at all. Twelve of 53 are CPU tasks. That is a direct, citable answer to the most common excuse for not starting: you do not need to rent an H100 to do real work in a serving engine, because a large fraction of the work is request handling, scheduling logic, state management and the public API surface, none of which require accelerator time to reason about or to test.

The second is that "model support" is a first-class family alongside runtime execution. A serving engine spends an enormous amount of its engineering effort on the unglamorous business of making a new architecture load, run and produce identical outputs to the reference implementation. That work is tractable for a newcomer in a way that rewriting an attention kernel is not, and it is genuinely valued, because it is what lets a platform team adopt a model the week it ships.

Here is the skill map I would give someone starting in September 2026, organised by what you can prove rather than by what you can claim.

Layer	What it actually involves	GPU needed?	How you prove it
Public API surface	Request and response schemas, streaming semantics, error codes, compatibility with existing client libraries, sensible defaults	No	A merged patch fixing a compatibility or error-handling bug, with a regression test
Request lifecycle and scheduling	Queueing, continuous batching, admission control, cancellation, fairness between callers, backpressure	Rarely	A reproduction script that shows a scheduling pathology, plus the fix
Persistent state	KV cache lifetime, prefix reuse, eviction, correctness of state shared across requests and across restarts	Sometimes	A test that fails before your change because state leaked between requests
Model support	Loading a new architecture, weight mapping, tokeniser edge cases, numerical parity against the reference implementation	Usually	Parity numbers against the reference, on a named model and hardware
Runtime execution	Kernel selection, memory planning, graph capture, quantised execution paths, multi-GPU placement	Yes	A benchmark run with a baseline, a variance estimate and a stated hardware configuration
Serving operations	Capacity planning, autoscaling, cost per served token, incident response on latency regressions	Yes, in aggregate	A written postmortem with before-and-after numbers

You do not need all six. You need genuine depth in one or two and enough literacy in the rest to know when your change touches them. The engineers who get hired for this are the ones who can say "my patch is in the scheduler, and here is why it does not corrupt cache state under concurrency", rather than the ones who can name the most components.

The market case, with its caveats attached

The demand case for this specialisation is strong, and it is also mostly built on aggregator estimates rather than audited payroll data. Treat the following the way you would treat any survey: directionally useful, individually uncertain, and worth quoting with the source attached.

On volume, India recorded approximately 290,000 AI job postings in 2025, with a projection of roughly 382,000 in 2026 — growth of around 32 per cent. We looked at what those postings actually ask for, and how Britain's demand profile differs, in our analysis of the India and UK hiring gap. On the supply side, demand for AI engineers has been reported rising at roughly 40 per cent year on year while the skilled talent pool grows only 15 to 20 per cent. If both figures are even approximately right, the gap compounds, and specialisations that are hard to fake widen fastest.

On pay, the commonly cited 2026 bands look like this. Every one of them is an aggregator estimate.

Market and level	Commonly cited band	Source character
India, entry level	Approximately Rs 2-7 lakh per annum	Aggregator and consultancy estimates, 2026
India, 4-6 years' experience	Approximately Rs 10-15 lakh per annum	Aggregator and consultancy estimates, 2026
India, senior architect or lead	Rs 25-50 lakh per annum and above	Aggregator and consultancy estimates, 2026
India, AI engineer average base	Approximately Rs 10 lakh per annum; typical range Rs 6-16 lakh	Glassdoor, May 2026
UK, senior AI engineer base	GBP 90,000-150,000	Commonly cited 2026 market ranges

Two caveats you should carry into any negotiation. First, none of these bands is specific to inference engineering; they are general AI engineering bands, and a serving specialisation sits inside them rather than beside them. Anyone quoting you a precise inference-engineering salary figure as of 2026 is extrapolating. Second, aggregator averages are heavily skewed by which companies self-report, so the average tells you less than the range does. If you want the mechanics of turning a range into an offer, we wrote that up separately in the India and UK pay benchmarking guide.

What you need before you start

This is a prerequisite checklist, not a curriculum. If three or more of these are missing, spend a month on them before touching a serving engine, because the alternative is a fortnight of confusion that teaches you nothing.

• Python, at the level of reading someone else's large codebase. Not writing scripts. Following a request through ten files, understanding async code, and reading type annotations you did not write.
• Concurrency literacy. You should be comfortable with the difference between concurrency and parallelism, what a queue does under load, what a deadlock looks like, and why shared mutable state is the enemy. This is the single most load-bearing prerequisite, because it is where the one-third rejection rate comes from.
• The transformer inference loop, conceptually. Prefill against decode, why the first token costs differently from the hundredth, what the KV cache is and why its size governs how many concurrent requests fit in memory. Our explainer on prefill-decode disaggregation is a reasonable way in, because it forces the two phases to be distinct in your head.
• Linux and the command line. Processes, ports, memory, logs, environment variables, and enough shell to script a reproducible run.
• Git at collaboration level. Rebasing, writing a commit message a maintainer will not have to interrogate, responding to review comments without force-pushing over the conversation.
• Basic statistics. Enough to know that one benchmark run is not a measurement, and that a five per cent improvement inside a ten per cent variance band is not an improvement.

Notice what is not on the list: CUDA, a graduate degree, and access to a cluster. All three help. None is a gate.

The learning path, in order

Stage one: read the serving loop before you run it

Pick one engine and read it properly. SGLang has a specific advantage as of September 2026, which is that the SWE-Serve tasks are derived from its merged pull requests, so the shape of real work in that repository is now documented in a paper you can cite. vLLM is the other obvious target, with a very large contributor base and a wide surface of approachable issues. Either is a defensible choice; contributing shallowly to both is not.

Spend your first week following a single request from the HTTP handler to the scheduler to the model executor and back. Write the trace down as you go, in your own words, as a file in your own notes repository. This artefact is more valuable than it sounds: it becomes the basis of your eventual write-up, and it is the fastest possible way to discover which parts of the system you cannot yet explain.

Stage two: run a server and measure it honestly

Now start something small and measure it. The point of this stage is not the number; it is building the reflex of always having a baseline.

# Stage 2 — stand up a server and take a baseline.
# Use a small model first. The goal is a reproducible measurement,
# not an impressive one.

git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"

# Record the exact environment. Do this every single time.
python -c "import torch, platform; print(platform.platform()); \
print('torch', torch.__version__, 'cuda', torch.version.cuda)" | tee env.txt
nvidia-smi --query-gpu=name,memory.total,driver_version \
           --format=csv | tee -a env.txt

# Launch the server, pinned to one GPU, with an explicit context length
# so the KV cache budget is a known quantity rather than a default.
python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-0.5B-Instruct \
  --port 30000 \
  --context-length 4096 \
  2> server.log &

# Baseline at concurrency 1. This number is almost useless on its own.
# Its job is to exist, so that every later number has something to beat.
python -m sglang.bench_serving \
  --backend sglang --port 30000 \
  --num-prompts 200 --request-rate 1 \
  | tee bench_c1.txt

Stage three: break it under concurrency

This is the stage that separates people who have used a serving engine from people who understand one. A single-stream benchmark measures latency on an idle machine, which is a condition that never occurs in production. The interesting behaviour — queueing, batching, cache pressure, tail latency, and the state bugs that SWE-Serve's end-to-end tests are designed to catch — only appears when many requests overlap.

Write your own small load harness rather than only using the built-in one. You will learn more, and the harness becomes a portfolio artefact in its own right.

# Stage 3 — concurrency sweep with percentile reporting.
# The shape of p95 as concurrency rises is the single most
# informative plot in serving work.

import asyncio, json, statistics, time
import httpx

ENDPOINT = "http://127.0.0.1:30000/v1/chat/completions"
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"


async def one_request(client, prompt):
    started = time.perf_counter()
    first_token_at = None
    body = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 128,
        "stream": True,
    }
    async with client.stream("POST", ENDPOINT, json=body, timeout=120) as r:
        async for line in r.aiter_lines():
            if line.startswith("data:") and first_token_at is None:
                first_token_at = time.perf_counter()
    end = time.perf_counter()
    return {
        "ttft_s": (first_token_at or end) - started,
        "total_s": end - started,
    }


async def sweep(prompts, concurrency):
    """Hold exactly `concurrency` requests in flight at all times."""
    queue = asyncio.Queue()
    for p in prompts:
        queue.put_nowait(p)
    results = []

    async with httpx.AsyncClient() as client:
        async def worker():
            while not queue.empty():
                prompt = await queue.get()
                results.append(await one_request(client, prompt))

        await asyncio.gather(*[worker() for _ in range(concurrency)])
    return results


def report(results, concurrency):
    ttfts = sorted(r["ttft_s"] for r in results)
    p = lambda q: ttfts[min(len(ttfts) - 1, int(q * len(ttfts)))]
    return {
        "concurrency": concurrency,
        "n": len(results),
        "ttft_p50_s": round(p(0.50), 3),
        "ttft_p95_s": round(p(0.95), 3),
        "ttft_mean_s": round(statistics.fmean(ttfts), 3),
    }


if __name__ == "__main__":
    prompts = ["Explain the KV cache in two sentences."] * 200
    rows = []
    for c in (1, 4, 16, 64):
        # Three repeats per level. One run is an anecdote, not a measurement.
        for repeat in range(3):
            res = asyncio.run(sweep(prompts, c))
            row = report(res, c)
            row["repeat"] = repeat
            rows.append(row)
    print(json.dumps(rows, indent=2))

Run that against your chosen engine, then run it again with one variable changed — a different context length, a different batching setting, a quantised checkpoint. Record the environment file alongside every result. If you want the wider throughput-and-latency playbook that this exercise sits inside, our vLLM throughput and latency guide covers the tuning knobs in production terms.

Stage four: fix something real

Now go to the issue tracker, and apply one filter above all others: can you reproduce it? An issue you can reproduce deterministically is worth ten issues that sound interesting. Reproduction is most of the work, it is the part maintainers most appreciate, and a well-written reproduction posted as a comment is itself a contribution that gets you noticed before you have written a line of the fix.

Start with correctness bugs, not performance optimisations. A correctness fix has an unambiguous test: it failed before, it passes now. A performance claim requires baselines, variance estimates and hardware disclosure, and a newcomer's first performance pull request is very often rejected not because the idea was wrong but because the measurement was not defensible. Earn the right to make performance claims by first establishing that your measurements can be trusted.

What this costs, in India and in the UK

The good news for anyone doing this from Chennai, Coventry, Bengaluru or Bristol is that the costed path is nearly identical, because GPU rental is almost universally billed in US dollars. As of 2026, published H100 hourly rates across providers run from roughly 2.89 to 11.06 US dollars per hour depending on provider, region and commitment. That spread is the most important number in your budget: the same experiment can cost four times as much on the wrong provider.

Here is a realistic 90-day practice budget, assuming you do the CPU-shaped work on the machine you already own and rent GPU time only when a question genuinely requires it.

Activity	GPU hours	At approximately $2.89/hr	At approximately $11.06/hr
Reading the codebase, tracing a request, writing tests	0	$0	$0
First server launch and single-stream baseline	4	Approximately $12	Approximately $44
Concurrency sweeps, three repeats per level	12	Approximately $35	Approximately $133
Reproducing a reported bug on GPU	6	Approximately $17	Approximately $66
Before-and-after measurement for the pull request	8	Approximately $23	Approximately $88
Total over 90 days	30	Approximately $87	Approximately $332

Thirty GPU hours across three months is a deliberate constraint, not a limitation. It forces you to do the thinking on CPU and spend accelerator time only on questions you have already framed precisely, which happens to be exactly the discipline the job requires. Builders in India should also check subsidised domestic GPU capacity before defaulting to a global provider, since publicly supported compute programmes have been changing the arithmetic; builders in the UK should check university and research-council schemes for the same reason. Rates and schemes both move, so verify current pricing at the point of use rather than trusting any figure in an article, including this one.

Proof of work: the three artefacts that actually convert

This is the part that determines whether any of the above turns into an offer. Assume, correctly, that the person deciding whether to interview you will not open your code. They will spend somewhere between forty seconds and two minutes on your profile. Everything below is about surviving those two minutes while still holding up when a staff engineer does eventually read the diff.

Artefact one: a merged pull request in a serving project

One merged patch in SGLang or vLLM outweighs a dozen personal repositories, for a reason worth internalising: it means somebody with a maintainer's incentives looked at your work and accepted responsibility for it. That is an external signal, and external signals are the only kind that survive scepticism.

Scope it small and finish it. A well-tested fix to a real defect is a stronger signal than an ambitious refactor that stalls in review for four months. Write the pull request description as though a hiring manager will read it, because one might: what was broken, how you reproduced it, what you changed, how the test proves it. If your best work is in a corporate repository you cannot show, our guide on open-source proof of work for a first AI engineering role covers how to build a public equivalent from scratch.

Artefact two: a reproducible benchmark run

A benchmark is only evidence if someone else can run it. That means a public repository containing the harness, the exact environment file, the raw result rows and the plotting script — not a screenshot of a terminal, and not a number in a bullet point.

The specific thing that makes a serving benchmark credible is the concurrency axis. Reporting one latency figure tells a reader nothing about the system. Reporting p50 and p95 time-to-first-token at concurrency one, four, sixteen and sixty-four, with three repeats at each level and the spread shown, tells them you understand what you measured. Add the hardware, the model, the context length and the date. The habits transfer directly from evaluation work, and if you have not built that muscle yet, the evals portfolio guide is the shortest route to it.

Artefact three: the written postmortem

This is the one almost nobody produces, and it is the one that converts. Write eight hundred words on a single problem you actually debugged: the symptom, your first hypothesis, why it was wrong, what you measured next, the root cause, the fix, and — this is the part that signals seniority — what you still cannot explain.

The admission of an unresolved edge is not a weakness in this format. It is the clearest available evidence that you were doing engineering rather than narrating a tutorial. Hiring managers have read a great many confident write-ups with no uncertainty in them, and they have learned what that usually means.

Then make all three scannable. A useful template, which fits in a profile project slot:

• The claim, in one sentence. "Fixed a cache-state leak between concurrent requests in [project], merged [month]."
• The evidence, in one sentence. "Regression test fails on the parent commit and passes on mine; reproduction script in the repository."
• The measurement, in one sentence with units. "p95 time-to-first-token at concurrency 64, measured on a single H100 over three repeats, moved from X to Y; run-to-run spread was Z."
• The limit, in one sentence. "Not verified on multi-GPU placement or on quantised execution paths."
• The link. One click to the diff, one click to the raw results.

Four sentences and two links. That is the format that survives a two-minute scan and still rewards a deep read.

The pitfalls that cost people offers

1. Optimising a kernel in isolation and calling it inference engineering. A faster kernel in a microbenchmark is a real achievement and an incomplete one. If it has not been integrated, run under concurrent traffic and validated end to end, you have demonstrated the part of the job that the SWE-Serve finding says is not where patches fail.
2. Benchmarking at concurrency one. The most common flaw in portfolio benchmarks. An idle-machine latency number describes a condition production never reaches, and an interviewer who serves models for a living will spot it in the first ten seconds.
3. Claiming a speedup with no baseline. "40 per cent faster" without a stated before-state, hardware, model, concurrency level and repeat count is not a claim, it is a mood. Worse, it invites the reader to assume the number is the best of several runs.
4. Ignoring the variance. Two runs of the same configuration will differ. If you have not reported the spread, you cannot know whether your improvement exceeds it, and a five per cent gain inside a ten per cent band is noise presented as progress.
5. Treating state as somebody else's problem. Persistent state across requests is named explicitly in what the SWE-Serve tasks test. Any change that touches caching, sessions or batching needs a test proving that request A cannot observe request B.
6. Spreading across five repositories. Maintainer trust accrues to people who keep turning up in one place. Five drive-by contributions to five projects read as tourism; four contributions to one read as a colleague.
7. Letting the proof live only in a repository. A repository is a filing cabinet. It does not appear when someone in Bengaluru or Manchester is searching for a serving engineer, and it does not explain itself to a non-reader.

A 90-day plan

This assumes roughly ten hours a week alongside a job, which is the realistic case for most people reading this. Compress it if you have more time; do not compress it by skipping the measurement discipline, because that is the part being assessed.

Period	Focus	Deliverable at the end	GPU hours
Days 1-14	Pick one engine. Trace a request end to end. Get the test suite running locally.	A written architecture trace in your notes repository	0
Days 15-30	Launch a server with a small model. Take a single-stream baseline. Record the environment.	A baseline result file and an environment manifest	4
Days 31-45	Build the concurrency harness. Sweep four levels, three repeats each. Plot p50 and p95.	A public benchmark repository with raw rows and a plot	12
Days 46-60	Triage the issue tracker. Reproduce two reported bugs deterministically. Post the reproductions.	Two reproduction comments on live issues	6
Days 61-75	Fix one correctness bug. Write the regression test first. Open the pull request and iterate on review.	An open pull request with a test and a clear description	4
Days 76-90	See the review through. Write the postmortem. Publish all three artefacts on a public profile.	A merged patch, a benchmark and an 800-word write-up, linked from one page	4

At the end of that, you will not be a senior inference engineer. You will be something more useful for the purpose of getting hired: a candidate who can point at externally validated evidence in a specialisation where almost nobody can, in a market where postings are reportedly growing faster than the pool of people qualified to fill them.

Where to go next

The unusual thing about this moment, as of September 2026, is that the definition problem has been solved for you. SWE-Serve's 53 tasks, drawn from merged production pull requests and split 41 GPU to 12 CPU, describe the territory. Its end-to-end rejection finding names the skill. The benchmark's own results — a best figure of 75 per cent mean pass@1 for the best-performing model-effort configuration across eleven models tested, on the authors' own harness — tell you that automated systems are competent here and not finished, which is precisely the condition under which a human who understands the failure modes remains worth hiring.

So pick the engine. Trace one request. Take one baseline. Break it under concurrency. Fix one real thing. Write down what happened, including the part you could not explain. Then put it somewhere a hiring manager can find it without knowing your name first.

That last step is the one people skip, and it is the one that decides whether any of the preceding work converts. The talent gap is real and reported across both markets. The specialisation is, for the first time, provable. What remains is making the proof visible.