SWE-Serve rejects a third of the patches other tests pass — and that is the whole story

What you need to know

• The rejection rate is the finding. End-to-end serving tests reject roughly one third of the patches that pass other evaluations. That is a statement about how we measure coding agents, not about any one model.
• The tasks are real. 53 repository-grounded tasks derived from merged SGLang production pull requests — work that shipped, reviewed by people who maintain an inference server for a living.
• The ceiling exists. The best-performing model-effort configuration reached 75% mean pass@1, across 11 models tested. Agents are not hopeless at inference engineering; they are just badly measured by function-shaped tests.
• It is open. Reference implementation runs with Harbor; the code sits at github.com/NVIDIA/swe-serve, and the paper is arXiv 2609.26777.

The benchmark, in one table

SWE-Serve — full title "SWE-Serve: Benchmarking Agentic Engineering for Production Inference Serving", by Jennifer Williams, Dave Farris, Jeff Farris and Jiantao Jiao — does something most coding benchmarks do not. It picks one production codebase that matters, mines the pull requests that were actually merged into it, and then asks whether an agent can reproduce that work in a way the running system accepts.

Dimension	What SWE-Serve does
Task count	53 repository-grounded tasks
Task source	Merged SGLang production pull requests
Hardware split	41 GPU tasks (H100) and 12 CPU tasks
Coverage	Six inference-engineering families, spanning model support, runtime execution and public APIs
What is tested	Correctness including integration, via end-to-end execution, persistent state and concurrent requests
Models evaluated	11
Best result	75% mean pass@1 for the best-performing model-effort configuration
Harness and code	Reference implementation run with Harbor; code at github.com/NVIDIA/swe-serve

A note on honesty, because it matters here: the paper's abstract says the tasks span six inference-engineering families, and it documents that span as covering model support, runtime execution and public APIs. It does not enumerate all six by name. We are not going to invent the missing ones for the sake of a tidier table, and you should be sceptical of any write-up that does.

Why end-to-end changes the verdict

The interesting part is not that a third of patches fail. It is which third, and why the other tests did not notice.

Most coding benchmarks are unit-test shaped. You hand the agent an issue, it edits some files, and a test suite runs in a container. If the assertions go green, the patch counts. That design is tractable, cheap and reproducible — and it is also a very narrow slice of what "working software" means when the software in question is an inference server.

Production inference work has three properties that unit tests are structurally bad at catching. There is persistent state: KV caches, scheduler queues, loaded weights, warm memory pools. There are concurrent requests: behaviour that is correct for one request in isolation and wrong once two arrive together. And there is integration across the serving stack: a change to model support that is clean in the abstract and breaks a public API contract two layers up.

SWE-Serve tests correctness including integration through end-to-end execution, persistent state and concurrent requests. Put differently, it starts the thing and drives traffic at it. And when you do that, roughly a third of the patches that other evaluations had already waved through stop working.

The measurement problem is the news

We have spent two years arguing about model rankings on benchmarks that, it turns out, disagree with the running system about a third of the time in this domain. That is not a small correction. It is the difference between "our agent closes most inference tickets" and "our agent closes most inference tickets, and then a human finds out on Friday which ones it did not."

This is the same shape of problem AI Tech Connect has covered before from the other end of the telescope. We looked at why agentic reasoning benchmarks still leave agents scoring in the thirties, and at the gap between benchmark performance and what enterprises actually observe in production. There is also the uncomfortable finding that agents can do the engineering competently and still fail the research when the evaluation asks a harder question than the one they optimised for. SWE-Serve is the infrastructure-flavoured instance of that same pattern, and it is the cleanest quantification of it we have seen.

It also puts the recent run of headline SWE-style scores into perspective. When an agent tops SWE-bench Verified in the mid-seventies, the honest reading is now: that is a genuine result on a genuine benchmark, and it tells you comparatively little about whether the same agent can safely touch a scheduler.

What this means for how you evaluate your own agent-written code

Here is the transferable lesson, and it does not require you to run SWE-Serve at all.

Your acceptance test must exercise the real integration surface. Not a local unit test. Not a mocked client. The surface where your system meets the world.

In practice, for a team in Bengaluru or Bristol shipping agent-written changes into a service, that means four concrete moves.

1. Boot the thing. If the change touches a long-running service, the acceptance gate should start the service and keep it running across the test. A patch that passes in a process that exits after one call has told you almost nothing about a process that lives for days.
2. Carry state across the test. Run a sequence, not a case. If your system has caches, sessions, connection pools or warm state of any kind, the second and third requests are where the interesting failures live.
3. Send requests concurrently. Even a crude two-client test catches a surprising share of the failures that serial tests miss. Concurrency bugs do not announce themselves; they wait for load.
4. Assert on observable behaviour, not internal shape. Response correctness, latency envelope, error rate, memory that comes back down. Those are the things your users experience. Internal assertions are proxies, and agents are unusually good at satisfying proxies.

That measurement is cheap, it takes an afternoon, and it is far more useful than another leaderboard argument. It also tends to change the conversation in the room, because a number from your own repository is very hard to wave away.

A hireable specialisation just became measurable

There is a career reading of this paper that is easy to miss.

Inference engineering has always been one of those skills that is obvious in the room and invisible on a CV. You either know why the throughput collapsed at batch size 48, or you do not. Until now there was no shared artefact that let someone demonstrate that knowledge to a stranger. Hiring managers fell back on proxies: which company you worked at, which framework you name-dropped, whether you could talk fluently about paged attention.

SWE-Serve changes that, quietly. The task families are documented. The repository is public. The hardware split — 41 GPU tasks on H100 and 12 CPU tasks — tells you exactly how much of this work needs expensive silicon and how much does not. Twelve of the fifty-three tasks are reachable from a laptop, which is a meaningful on-ramp for anyone who cannot casually rent an H100 by the hour.

For builders in India and the UK, that on-ramp matters for different reasons and lands in the same place. Indian teams are increasingly the ones operating inference for products serving tens of millions of users on cost structures that leave no room for waste — a theme we dug into in our piece on inference cost economics. UK teams, particularly in fintech and health, are running smaller fleets under tighter regulatory scrutiny, where a patch that quietly breaks under concurrency is a governance incident rather than a latency blip. Both markets are hiring for the same skill and struggling to recognise it.

If you want the practical version of that argument — which of the documented families to study first, what to build to prove it, and how to talk about it in an interview — we have published a companion guide on breaking into inference engineering using the SWE-Serve skill map.

What the 75% actually tells us

It would be easy to file this paper under "agents are worse than you think" and move on. That is not the right reading either.

Seventy-five per cent mean pass@1 for the best-performing model-effort configuration, on tasks derived from merged production pull requests in one of the harder corners of the stack, is a serious result. These are not toy issues. Someone senior wrote each of these changes, someone else reviewed it, and it went into a system that other people depend on. An agent reproducing three quarters of that work under an honest end-to-end gate is not a party trick.

So both things are true at once. The capability is real, and the measurement we have been using to describe it has been flattering it by roughly a third in this domain. The correct response is not to trust agents less. It is to test them properly and then trust the result — which, unusually for this field, is a fully actionable conclusion.

What to do this week

• Read the abstract yourself. arXiv 2609.26777. It is short, and the methodology section is the part worth your time.
• Look at the repository. The reference implementation runs with Harbor; the task definitions at github.com/NVIDIA/swe-serve are a usable template for how to write an end-to-end gate of your own, whatever your stack.
• Measure your rejection rate. Thirty recent agent-written patches, one real end-to-end gate, one number. That number is your actual readiness, and nobody else can produce it for you.
• Write down what you find. If you are a builder who has fixed a concurrency or state bug that a test suite missed, that story is worth more on a profile than any framework list.

The industry has been grading coding agents on whether the tests go green. SWE-Serve is a reminder that the tests going green and the server still serving are two different claims — and for the last couple of years we have mostly been reporting the first one while implying the second.