ProgramDistill Mines 4,063 Coding Tasks From 26 Live Web Apps

What changed this week

• A new benchmark and a new pipeline. "ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks" (arXiv 2609.18805, September 2026) comes from KAIST, Microsoft Research Montreal and Microsoft AI. A dataset is published at huggingface.co/datasets/microsoft/ProgramDistill.
• Scale without labellers. The pipeline discovered 1,975 replay-verified behaviours across 26 applications and constructed 4,063 tasks without human intervention.
• Three difficulty bands. Atomic repair, cumulative repair, and full-application reconstruction — a built-in ladder rather than one undifferentiated pile of problems.
• Nine frontier coding agents were evaluated. On cumulative workflows within the full-application-reconstruction band, GPT-6 Astra reached 49.2% success and Claude Opus 5 reached 28.8%.
• The authors' framing. A scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and a natural basis for future curriculum-based training.

Read that list again and notice what is missing: no crowd of annotators, no hand-written test suites, no committee arguing about whether a task is fair. The tasks were mined from applications that already work. That is the whole trick, and it is why this paper is worth an hour of your team's time even if you never download the dataset.

How mine-craft-patch works

The pipeline has a deliberately plain name: mine-craft-patch. Three stages, each doing one job.

Mine. Drive a live, working web application through a browser and record reproducible interaction traces. A trace is not a screenshot or a vague description; it is a sequence that can be replayed and is expected to produce the same observable outcome every time. Traces that do not replay cleanly are discarded. What survives is a catalogue of behaviours the application demonstrably has — 1,975 of them across 26 applications in the published run.

Craft. Take the application's source and mask an implementation — remove the code responsible for some slice of behaviour, leaving the surrounding structure intact. The masked application plus the recorded trace becomes a task: here is a codebase with a hole in it, and here is the behaviour that used to come out of that hole.

Patch. Hand the masked application to a coding agent and let it work until the behaviour is restored. The recorded trace, replayed against the agent's patched build, is the judge. Either the interaction produces the reference outcome or it does not.

The load-bearing property is that every task is replay-verified before any agent sees it. The reference application performs the behaviour, so the task is known to be solvable in that codebase. A failure is therefore information about the agent, not a coin flip about whether the task was possible at all. Anyone who has watched an evaluation harness produce a 40% score and had no idea how much of the missing 60% was impossible will understand why that matters.

The three bands, read as a maturity ladder

The most immediately reusable part of ProgramDistill is not the tasks but the way difficulty is organised. Three bands, increasing in how much state and coordination a solution requires.

Band	What the agent is asked to do	What it tells you about an agent	Use it internally when
Atomic repair	Restore one masked implementation so a single verified behaviour works again	Baseline competence: can it read the surrounding code and write something that fits?	You are choosing between agents or models at all, and need a floor
Cumulative repair	Restore behaviour across a sequence where earlier work carries into later steps	Whether quality holds as state accumulates, or degrades step by step	You are about to let an agent touch multi-file changes unsupervised
Full-application reconstruction	Rebuild the missing functionality of an application against the live reference	Whether the agent can hold a whole system in view rather than a single function	Someone senior is proposing "the agent builds the service"

Most teams I speak to, in Bengaluru and in London alike, are running an informal version of band one and then extrapolating straight to band three in their planning documents. The bands exist precisely because that extrapolation does not hold.

The two numbers, and exactly what they mean

Nine frontier coding agents were evaluated. Two of those results have been quoted widely since the paper appeared, and they are easy to repeat without their scope attached, so here they are with it.

Agent	Reported success	Exact scope of this number
GPT-6 Astra	49.2%	Cumulative workflows within the full-application-reconstruction band only. Not an overall ProgramDistill score.
Claude Opus 5	28.8%	Cumulative workflows within the full-application-reconstruction band only. Not an overall ProgramDistill score.

What the pair of numbers does support is a narrower and more useful claim: the higher of the two widely-quoted results in that slice leaves roughly half the cumulative workflows unfinished. If your internal business case assumes an agent reconstructs application functionality across a long sequence of dependent steps and lands it, the figures that have circulated for that band sit under fifty per cent. The paper evaluated nine agents in total, and the remaining seven results are not part of the pair quoted here. Plan for review, not for autonomy.

Why agents fall apart as state accumulates

The gap between atomic repair and cumulative workflows is the part that matches what teams actually see in production. An agent handed one well-scoped function with clear neighbours tends to do well. The same agent, twelve steps into a sequence where step three introduced a small inconsistency, tends to compound it. Each step inherits the previous step's output as context, so a minor early error is not corrected — it becomes the premise for everything after.

This is why single-shot benchmark scores flatter agents so badly. A one-shot task resets the world for every attempt. A real sprint does not. The cumulative bands are closer to the shape of the work, and that is where the numbers drop.

It is the same lesson the SWE-Serve benchmark arrived at from a different direction — that a patch passing a unit test says far less than a system still serving traffic afterwards. And it is why the recent work on open agent trajectories matters: the trajectory, not the final diff, is where the failure is legible.

What this means for teams buying or building coding agents

Here is the uncomfortable position most engineering organisations are in. A product team in Bengaluru is choosing between three agent vendors. A fintech platform team in London is deciding whether to let an agent open pull requests against a regulated codebase. A GCC engineering organisation in Hyderabad or Pune is being asked by its parent company to report an "AI productivity" figure. None of them has a way to measure whether any of these agents works on their code. So they read a public leaderboard and hope.

ProgramDistill's contribution is that the loop is copyable at small scale. You do not need 26 applications or 4,063 tasks. You need one application you own, a handful of interactions that replay reliably, and the discipline to mask an implementation rather than invent a synthetic exercise.

1. Pick a reference build. A tagged release that passes its end-to-end suite. This is your ground truth; everything else is measured against it.
2. Record traces, not descriptions. Ten to twenty end-to-end interactions that cover the behaviour you care about and replay deterministically. Discard flaky ones ruthlessly — a flaky trace poisons every score built on it.
3. Mask, do not invent. Delete a real implementation and keep the surrounding code. Writing a synthetic exercise reintroduces the very bias the paper removed.
4. Build the ladder. One masked module is atomic. Three dependent modules restored in sequence is cumulative. A whole feature area is your reconstruction band.
5. Report bands separately. Never average across bands into one headline figure — that is how a 49.2% turns into a claim nobody can defend.

If you want a worked version of this discipline applied to your own portfolio rather than your employer's codebase, our guide on proving you can build evals covers the same loop at individual scale, and the companion piece on judging GUI agents by trajectory is directly relevant given that ProgramDistill's verification signal is a replayed browser interaction.

The honest caveats

Three things worth holding onto before anyone builds a strategy deck on this paper.

The dataset is published, which is not the same as open-source. A dataset exists on Hugging Face under the Microsoft organisation. That tells you where to find it; it does not tell you what you are permitted to do with it, and the applications it was distilled from carry their own terms. Read the licence on the dataset card yourself before you evaluate against it, and certainly before you train on it.

Web applications are one shape of software. The mining step depends on a browser-driven interaction surface. Teams building data pipelines, embedded firmware or batch systems can borrow the principle — record a verified behaviour, mask the implementation, check restoration — but the specific harness will not transfer unchanged.

Replay verification checks observable behaviour, not quality. An agent can restore the trace with code you would reject in review. Replay tells you the behaviour is back. It does not tell you the patch is maintainable, secure, or anything you would want to live with. Keep a human in that loop. For teams running several agents behind one interface, the orchestration question raised by multi-model orchestration in Copilot CLI becomes sharper here: per-band measurement is how you decide which agent gets which class of task.

Where this is heading

The authors position ProgramDistill as a natural basis for curriculum-based training — start an agent on atomic repairs, progress to cumulative workflows, then to reconstruction. That is a reasonable direction and an obvious one once you have a pipeline that generates graded, verified tasks at scale without people.

For everyone else, the useful takeaway is smaller and available now. The bottleneck in evaluating coding agents was never compute or model access. It was that constructing honest, solvable, verifiable tasks required human effort nobody wanted to fund. ProgramDistill shows that a working application is itself a source of such tasks, for free, if you are willing to record what it does and then take pieces away. Whether you are a four-person team in Bristol or a platform group in Chennai, that is a method you can run this quarter.

The paper is arXiv 2609.18805. Read the difficulty-band definitions carefully before you quote any percentage from it.