Agents Did the Engineering and Still Failed the Research

The method is the contribution

Most claims about agents automating research collapse under one of two problems. Either the evaluation uses narrow, verifiable tasks — which by construction excludes the open-ended part that makes research hard — or it submits AI-generated papers to blind peer review, a process the authors describe bluntly as overstretched, stochastic and suffering from poor review quality.

The paper, Can AI agents conduct open-ended AI research? Early evidence from two case studies (arXiv 2607.27191), proposes a third option it calls a shadow evaluation. The design is elegant:

1. Take a high-quality paper that has been written but not published.
2. Give an agent the paper's central, open-ended research question.
3. Let it work with real time and real compute.
4. Have the paper's original authors — the only people on earth who know what a good answer looks like — grade the output.

That last step is what makes it work. The graders are not generic reviewers applying a rubric. They are the people who already solved the problem and know exactly which turns were productive and which were dead ends.

Study parameters, as reported in arXiv 2607.27191 (submitted 29 July 2026).

Parameter	Value
Papers evaluated	2 unpublished NeurIPS 2026 submissions
Time per agent	Six days
Compute budget	Thousands of dollars
Human assistance during the run	None
Engineering completed	All of it
Research progress	Not substantial
Author verdict	Both unambiguously rejected
Robustness check	Second model and scaffold reproduced the failures

The authors have released the expert reviews, survey responses, agent repositories and logs. That matters more than the headline: a negative result with published artefacts is something other people can argue with, which is the point.

Be honest about the sample size

Two papers is two papers. Arvind Narayanan, one of the authors, said publicly on posting that the results are tentative and that the team is working to address limitations including sample size and potential scaffold improvements. That caveat is in the paper too — the title says "early evidence" and means it.

The five failure modes, and what each one means in practice

Read these as a description of what your own agents do on long tasks, because they will be familiar.

Poor judgment about the bar for publishable research

The agent could not tell the difference between a result and a good enough result. It produced output, considered the task complete, and was wrong about that. In production terms this is the agent that closes a ticket having technically satisfied the literal request while missing the standard the request implied.

Uncreative responses to shortcomings in the research design

When the initial approach had a flaw, the agent patched around it rather than reconsidering the approach. Research progress usually requires noticing that the question was framed wrongly. Patching preserves the frame.

Ineffective backtracking from dead ends

This is the expensive one. The agents pursued unproductive directions past the point where a researcher would have abandoned them. Sunk cost is not a human weakness here — it is a structural consequence of a system whose context contains a great deal of work already invested in an approach.

Poor resource awareness

Given six days and a real compute budget, the agents did not allocate either well. They had no reliable sense of what an experiment would cost before running it, or how much of their budget remained relative to how much of the problem was left.

Instruction drift

Over long horizons the agents moved away from what they were originally asked to do. Anyone who has watched a coding agent begin with a bug fix and end up restructuring a module has seen this. We wrote about a related effect when covering the AGENTS.md ablation study: instructions placed in context are not the same thing as instructions being followed at hour forty.

Every one of these is a harness problem

Here is the reframe that makes this paper useful rather than merely interesting. Four of the five failure modes are things a well-built system can compensate for. They are not mysterious cognitive deficits; they are consequences of running a model in a loop with no external structure.

Mapping each failure mode to a concrete engineering countermeasure.

Failure mode	What the harness should do instead
Poor judgment about the bar	Define the acceptance criteria before the run, as a checkable artefact. If a human cannot state the bar, an agent cannot meet it.
Uncreative design fixes	Force a periodic "is the framing wrong?" checkpoint with a fresh context that has not seen the accumulated work.
Ineffective backtracking	Set explicit abandonment triggers — n failed attempts, or x per cent of budget with no measurable progress — and make abandoning a first-class action rather than a failure.
Poor resource awareness	Expose the remaining budget in context on every turn, in the same units the agent spends. Hidden budgets cannot be reasoned about.
Instruction drift	Re-inject the original objective verbatim at intervals, and diff current activity against it. Do not rely on it surviving in a compacted history.

Several of these are things we have covered as standalone practice. Periodic re-grounding and objective preservation are core to context engineering for long-horizon agents, and explicit state transfer at boundaries is the subject of our guide to agent handoff contracts. What this paper adds is empirical grounding: these are not stylistic preferences, they are the specific places a six-day autonomous run comes apart.

What this says about the automation timeline

The framing in the abstract is precise and worth quoting on: today's agents "can do the engineering of AI research, but struggle with critical parts of the research lifecycle."

That is not a small claim in either direction. Completing all the engineering for a NeurIPS-calibre research question, unaided, in six days, would have been science fiction three years ago. It is genuinely remarkable and the paper says so. What did not happen is the part that forecasts of explosive recursive progress depend on — an agent taking a hard open question and moving it forward.

The distinction matters because the two capabilities have very different economic consequences. Automated engineering compresses the cost of executing an idea. Automated research would compress the cost of having good ones. Only the second changes the shape of the curve, and this study finds no evidence of it yet.

What this means for your career, said directly

If the engineering is the automatable part and the judgement is not, then the thing worth being able to demonstrate is judgement. That sounds like career-advice filler until you ask what evidence of judgement actually looks like, and notice that most engineers have none.

A repository shows that you can build. It does not show that you knew when to stop, that you noticed the framing was wrong, or that you killed a promising-looking direction on day three because the numbers were not moving. Those are exactly the five things the agents could not do, and exactly the things nobody writes down.

The fix is not complicated. Alongside the work itself, write the account of the decisions: what you tried first, why it was wrong, what made you abandon it, what the bar was and how you knew you had cleared it. Our guide to writing a hiring-ready case study covers the structure. The reason this converts in interviews is not that hiring managers enjoy reading process notes. It is that a candidate who can articulate a good abandonment decision has demonstrated the one capability the current generation of agents visibly lacks.

There is a related version of this for teams building evaluations. If your eval suite only measures whether the code ran, it is measuring the automated half. Measuring the other half is harder and more valuable, which is the argument in our guide to building evals agents cannot game.

The short version

Twenty-three authors gave frontier agents six days and thousands of dollars on the central questions of two unpublished NeurIPS 2026 papers, then had the original authors grade the results. The agents completed every piece of engineering without help and made no substantial research progress; both were unambiguously rejected, and a second model with a different scaffold failed the same way. Five failure modes — judgement about the bar, uncreative design fixes, poor backtracking, poor resource awareness, instruction drift — explain the gap, and four of the five are fixable in your harness today. The sample is two papers and the authors are appropriately cautious about it. The durable takeaway is not a verdict on agents. It is a list of the specific places long autonomous runs break, and a fairly direct answer to what human AI engineers should be able to prove they can do.

The preprint, with released reviews, logs and repositories, is at arxiv.org/abs/2607.27191.