Final Scores Lie: How to Evaluate Long-Horizon Agents

What you need to know

• One number is not an evaluation. Two agents that both score 40 per cent on a long-horizon task can be failing for entirely different reasons, and those reasons need entirely different fixes.
• Variance is a first-class metric. Running an eval once and reporting the result is the single most common mistake in agent evaluation as of September 2026.
• Behaviour is measurable. Solution framing, execution and feedback control can all be instrumented with rule-based signals you already have in your logs.
• Curves beat endpoints. Recording a progress metric per step turns each run into a shape, and the shape tells you which layer to fix.
• Harness is a confound. Reports that "the model got worse" frequently turn out to be harness regressions that were never attributed correctly.

In August 2026 a group of researchers from Meituan and the University of Chinese Academy of Sciences published Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv 2608.13417, 13 August 2026). They evaluated seven frontier models on 36 long-horizon tasks, and instead of ranking them on a leaderboard they built a framework of rule-based metrics that characterise what happens inside a run, across three axes: Solution Framing, Execution and Feedback Control. They added controlled comparisons to assess how well agents reuse experience both within a single task and across tasks.

Their headline conclusion is uncomfortable for anyone selling autonomous research agents. Current agents, the paper finds, behave more like engineering optimisers than autonomous researchers. They can formulate and implement practical solutions. But performance varies substantially across runs on the same task, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty is rare. Performance is shaped by distinct process bottlenecks, by experience reuse that can either help or mislead later decisions, and by harness design that affects stability.

Every one of those findings is a direct instruction about how to run your own evaluations. This guide turns them into a method.

What a final score actually hides

Consider an agent asked to take a failing data pipeline, diagnose the cause, patch it, and get a test suite green. That is thirty or forty decisions: which file to open first, whether to read the logs or the code, whether to trust the error message, whether the first fix worked, whether to roll it back. At the end you record one number. Pass, or fail. Or, if you are being generous, 40 per cent of subtasks completed.

That number is a lossy compression of the entire run, and the loss is not random — it discards exactly the information you need to improve anything. Two agents that both land on 40 per cent might have diverged like this:

• Agent A framed the problem correctly in step two, executed cleanly for fifteen steps, then ran out of context at step eighteen and never recovered. That is a harness and context-management problem. The reasoning was fine.
• Agent B misread the failure in step one, committed to the wrong hypothesis, and spent thirty steps making confident, well-executed changes to the wrong module. That is a framing problem, and no amount of extra context window will fix it.

Same score. Opposite fixes. If your evaluation reports only the score, you will spend a sprint on the wrong one. The question that matters for a long-horizon run is not "did it finish" — it is "where did it stop making progress, and did it notice".

That second clause is the one teams forget. An agent that fails but detects the failure and reports it is operationally very different from one that fails and confidently declares success. Both score zero. One of them is safe to put in front of a customer behind a human review step; the other is not.

Run variance is the first thing to measure

Most teams run an eval once. On a short classification task that is defensible — the output distribution is narrow and one sample is roughly representative. On a long-horizon task it is not, because every step is a branch point and the branches compound. The arXiv 2608.13417 authors state it plainly: performance varies substantially across runs on the same task.

So the first change to your harness is not a new metric. It is a loop. Run each task N times, and report the distribution.

Why the spread matters more than the mean

Imagine two candidate agents, and treat these figures purely as an illustration rather than a measurement. Agent X averages 55 per cent across ten runs, but the runs range from 10 per cent to 95 per cent. Agent Y averages 45 per cent, with every run between 40 and 50 per cent. A leaderboard picks X. Production usually wants Y.

The reason is that in production you do not get the mean. You get individual runs, and you pay the operational cost of the bad ones — the engineer who has to unpick a half-finished change, the customer-facing incident, the review queue that suddenly needs three people instead of one. That cost is rarely linear in the score. A run at 10 per cent that left a repository in a broken state can cost more than five clean successes save. When your tail is expensive, a stable mediocre agent beats an erratic strong one, and the mean hides exactly this.

The same logic is why teams shipping model upgrades increasingly stage them rather than cutting over, an approach worth reading alongside shadow and canary deploys for LLM model upgrades. A canary exposes the tail to a small slice of real traffic before the tail finds you at full volume.

How many repeats is enough

There is no clean statistical answer here, and you should be suspicious of anyone who offers one with three runs of data. Some honest heuristics:

• Three repeats tells you almost nothing. It will catch a catastrophically broken agent and nothing subtler.
• Five repeats is a practical floor. It is usually enough to notice that a bad tail exists, which is the finding that changes decisions.
• Ten or more starts to give a usable picture of the spread on a task you genuinely care about.
• Fewer tasks, more repeats beats many tasks run once, when the decision is "should we ship this". Broad single-run coverage is for regression detection, not for release decisions.

Report the repeat count next to every figure. A team that sees "68 per cent (n=3)" will read it correctly; a team that sees "68 per cent" will treat it as a fact. And resist the temptation to compute a confidence interval from five runs and present it as though it settles an argument — with that many samples you are describing a rough interval, not demonstrating a significant difference. Say "the spread was wide" rather than dressing it in notation it cannot support.

Instrumenting the three axes

The three axes in arXiv 2608.13417 — Solution Framing, Execution and Feedback Control — are the useful part of the paper for practitioners, because they are not vibes. They decompose the run into behaviours you can count with rules, without needing a judge model at all.

Here is how to turn each axis into per-run signals your harness can log. None of these requires anything you do not already have in a trace.

Axis	What it asks	Per-run signals to log	What a bad reading means
Solution framing	Did the agent form a plan, and was it the right shape?	Did it state an explicit plan before acting; at which step; how many distinct hypotheses it named; how many times the plan changed and at which steps; whether any change followed new evidence or none; step index of the first irreversible action	Late or absent framing, or a plan that never changes despite contradicting evidence. Usually a prompt or scaffold problem, not a model capability problem.
Execution	Was the work itself competent?	Tool-call error rate; malformed-argument rate; retry loops and their depth; wasted steps (no observable state change); steps spent re-reading material already in context; ratio of edit steps to verification steps	High wasted-step counts mean the agent is burning horizon without moving. Often a tool-description or tool-ergonomics problem.
Feedback control	When something failed, did behaviour change?	Consecutive non-improving steps before a strategy change; whether a failing test produced a different action or a repeat of the same one; number of identical or near-identical tool calls in a window; whether the agent ever declared the task blocked; whether its final self-assessment matched the ground truth	Repeating an identical failing action is the clearest signal of no feedback control. A final self-assessment that disagrees with ground truth is the silent-failure signal.

The last row of that table is the one to instrument first if you only have time for one. Consecutive non-improving steps before a strategy change is a single integer, it is cheap to compute, and it correlates with almost everything teams complain about when they say an agent "gets stuck".

These axis metrics are deliberately rule-based. Where you do need a model to grade something subjective — the quality of a written plan, say — calibrate that judge against humans first, because an uncalibrated judge will happily manufacture a trend. The method is in calibrating an LLM judge against human agreement, and it applies just as much to grading a plan as to grading an answer. For agents whose steps are screenshots rather than tool calls, the trajectory-grading variant is covered in judging GUI agents on screenshots and trajectories.

Progress curves instead of endpoints

This is the single most useful change you can make, and it is embarrassingly simple: define a task-specific progress metric, record it after every step, and plot it.

A progress metric is whatever moves monotonically towards done for that task. Common choices:

• Tests passing out of a fixed suite — the default for engineering tasks.
• Sub-goals met against a checklist written before the run.
• A rubric score from a calibrated judge, for tasks with no mechanical check.
• A domain metric — rows reconciled, documents classified, a validation figure on a held-out split.

Record it per step with the axis signals alongside, and a run stops being a number and becomes a curve. Here is an illustrative record shape — the field names are a suggestion, not a standard, and the values are invented to show the structure.

// ILLUSTRATIVE ONLY — example values, not measurements
{
  "run_id": "rca-pipeline-fix/model-a/seed-03",
  "task_id": "rca-pipeline-fix",
  "step": 17,
  "timestamp": "2026-09-22T11:04:18Z",

  "progress": {
    "metric": "tests_passing",
    "value": 6,
    "max": 14,
    "delta_from_prev_step": 0
  },

  "action": {
    "type": "tool_call",
    "tool": "run_tests",
    "args_hash": "b21f0c",
    "duration_ms": 8420,
    "ok": true
  },

  "framing": {
    "plan_stated": true,
    "plan_version": 2,
    "plan_changed_this_step": false,
    "active_hypothesis": "stale schema cache"
  },

  "execution": {
    "state_changed": false,
    "repeat_of_previous_action": true,
    "retry_depth": 2,
    "wasted_step": true
  },

  "feedback_control": {
    "steps_since_progress": 5,
    "strategy_changed_this_step": false,
    "agent_declared_blocked": false
  },

  "harness": {
    "context_tokens": 148230,
    "context_truncated": false,
    "tool_timeout": false,
    "rate_limited": false,
    "retries_exhausted": false
  }
}

Note that harness is a sibling of the behaviour blocks, not mixed into them. That separation is what makes the next section possible.

Reading the shapes

Once you have curves, you stop arguing about scores and start recognising patterns. Four shapes cover most of what you will see.

Curve shape	What it looks like	Most likely root cause	Where to fix it
Early plateau	Rises for two or three steps, then flat for the rest of the horizon	Wrong framing. The agent committed to a hypothesis that cannot solve the task and never revisited it.	Prompt and scaffold: force an explicit plan, require it to name an alternative hypothesis, add a mandatory re-plan trigger after N non-improving steps.
Late collapse	Steady climb, then a sharp drop near the end of the run	Context or memory. Something the agent needed was compacted away, or a summarisation step dropped a constraint it had been honouring.	Context management: pin the task statement and acceptance criteria outside the compactable region, checkpoint state externally, log every truncation event.
Sawtooth	Repeated rise and fall — progress made, then undone, then remade	Tooling. Non-idempotent operations, an edit tool that clobbers, a test runner with side effects, or two actions that fight each other.	Tool layer: make writes idempotent, add a diff-preview step, and give the agent a way to check current state cheaply before acting.
Steady climb, short	Clean progress that simply stops when the step budget or wall-clock limit is reached	Harness limits, not capability. The agent was working and you switched it off.	Harness: raise the step budget for that task tier, or split the task. Do not touch the prompt — there is nothing wrong with it.

That last row deserves emphasis because it is so often misdiagnosed. A truncated steady climb gets logged as a failure, the team concludes the agent "cannot do multi-step work", and someone spends a fortnight rewriting prompts for a problem that was a configuration value. Curves make it obvious in a glance.

Sawtooth curves, meanwhile, are usually a durability problem in disguise — the agent is fighting a substrate that does not survive retries cleanly. The patterns for fixing that layer are in durable agent execution with checkpoints, resume and idempotency, and fixing the substrate frequently moves the curve without touching the model at all.

Separating agent failure from harness failure

The paper is explicit that harness design affects stability. This is the most under-reported confound in agent evaluation, because the harness is usually built by the same team running the eval, changes silently between runs, and does not appear in any report.

The discipline is ordinary experimental hygiene, applied to a layer people forget is a variable:

1. Hold the harness constant and swap the model. Same scaffold, same tools, same step budget, same context policy, same seeds where you control them. Only then is a difference attributable to the model.
2. Hold the model constant and swap the harness. Run the previous harness version against the current model. If the numbers move, your harness change is the story — and you have just found a regression you would otherwise have blamed on the provider.
3. Version the harness like code. Every eval result carries a harness version string. If you cannot say which harness produced a number, that number is not comparable to any other number.
4. Log harness events as their own type. Context truncation, tool timeout, rate limit, sandbox restart, retry exhaustion, step-budget exhaustion. Never let these land in the same bucket as model decisions.

Rule four is the highest-leverage. Once harness events are distinct, your failure taxonomy splits cleanly into "the agent decided badly" and "the environment stopped it", and those two columns almost never trend together. As of September 2026, a reasonable working assumption when someone reports that a model has degraded is that the harness changed first — check that before opening a support ticket.

A related trap is qualifying a target before you have a stable harness to qualify it against. If you are choosing what the agent should be allowed to do at all, the qualification-first approach in setting an agent reliability target and qualifying the harness pairs directly with everything in this section.

Experience reuse can mislead

One of the more interesting parts of arXiv 2608.13417 is its use of controlled comparisons to assess experience reuse, both within a single task and across tasks. The finding is not a clean win: reused experience can either help or mislead later decisions.

That is worth sitting with, because memory is currently sold as a straightforward upgrade. Give the agent a store of what it learned last time, and it gets better. Sometimes. And sometimes it carries forward a heuristic that was true for one task and false for the next, applies it with more confidence than it applied anything on the first task, and fails in a way that is harder to debug because the cause is in a memory store rather than the visible trace.

So test it, rather than assuming it:

• Run the same task set twice — once with prior-task memory enabled, once cold — and compare distributions, not means. Memory often helps the median and hurts the tail, which is precisely the pattern a mean conceals.
• Vary task order across seeds. If memory genuinely helps, the benefit should survive reordering. If it only helps in one order, you have measured a coincidence.
• Log every memory read with the step that consumed it. When a run fails, you want to know which stored lesson was in context at the moment of the wrong turn.
• Date and expire stored lessons. A heuristic learned against last quarter's schema is actively harmful against this quarter's, and nothing in the agent will tell you it has gone stale.
• Distinguish within-task from across-task reuse. Remembering "the config lives in /etc/app" within one task is almost always useful. Carrying "the config lives in /etc/app" into a different service is how a confident wrong answer gets manufactured.

Building a long-horizon eval set you can afford

Everything above costs money. A thirty-step run with a large context is not cheap, and the whole argument of this guide is that you should be running each task five or ten times rather than once. Multiply those together and a naive implementation produces a bill that gets the programme cancelled.

The answer is tiering. Run a small set of full-length tasks often, and a larger set of truncated ones constantly.

• Tier 1 — full-length, small, frequent. Five to ten tasks that genuinely represent the product, run at full horizon with five or more repeats each, on every model change and every significant harness change. This tier produces the numbers you make decisions with.
• Tier 2 — truncated, larger, constant. Thirty to a hundred tasks capped at a fraction of the full horizon — a third is a reasonable starting point — run on every commit with one or two repeats. This tier is a smoke alarm, not a verdict.
• Tier 3 — adversarial, occasional. A handful of tasks built specifically around the failure shapes you have already seen, run before releases. This tier stops regressions you have paid for once from being paid for twice.

When a truncated result can be trusted

A truncated run is trustworthy for a negative and unreliable for a positive. If the agent has made no progress by the truncation point, that is real information — early plateau shows up early by definition. If the agent is climbing steadily at the truncation point, you have learned that it has not failed yet, which is not the same as learning that it will succeed. Late collapse lives, by definition, past the cut.

So: let Tier 2 fail a build, never let it pass a release. And set the truncation point from data rather than taste — look at your Tier 1 curves, find the step index by which successful runs have typically reached most of their progress, and cut a little after that.

Estimating what a cycle costs

Do not budget from a vibe, and do not copy someone else's figures — token prices, task lengths and step counts all differ too much between teams for a borrowed number to mean anything. Measure your own with a straightforward method:

1. Instrument one full-length run end to end and record input tokens, output tokens, tool-execution time and wall-clock time. Your provider's usage API gives you the first two per call.
2. Multiply by your own current per-token rates to get cost per run. Include cached-read rates separately if your harness reuses a stable prefix, because on long runs that line dominates.
3. Multiply by repeats per task, then by tasks per tier, then by cycles per week. That product is your weekly eval spend, and it is usually the first time anyone on the team has seen it written down.
4. Divide the Tier 1 total by the number of shipping decisions it informs per month. Cost per decision is the figure that survives a conversation with a finance team; cost per run is not.

Once you have that model you can trade explicitly — fewer tasks with more repeats, or more tasks with fewer — instead of discovering the trade after the invoice. The broader framing for that conversation, including how to express eval spend as quality per unit of currency, is in cost-aware evals and quality per pound.

A reporting template that survives contact with stakeholders

None of this matters if the output of your eval is still a single percentage in a deck. The report is where the method either lands or evaporates. Five things belong in it.

1. Distribution, not mean. Per task: median, worst run, best run, and the repeat count. If you must give one headline number, give the worst run, because that is the one the business will eventually experience.
2. Curve-shape mix. How many runs were early plateau, late collapse, sawtooth, truncated climb, or clean success. This single row tells a technical reader what to work on next week.
3. Failure taxonomy counts. Categorised by the three axes — framing failures, execution failures, feedback-control failures — with a count each. Counts, not anecdotes.
4. Harness-attributed failures, separately. Context truncations, timeouts, rate limits, budget exhaustions, broken out and excluded from the model's score. If this column is large, the report is about your infrastructure, not the model.
5. Delta versus the last version. Model version, harness version, and the change in each of the above. Never present a delta across a change in both, and say so explicitly when you cannot isolate one.

Two lines of discipline make the whole report credible: every figure carries its repeat count, and anything that cannot be traced to per-step logs does not go in at all. For a regulated-sector supplier presenting agent results to a client's risk committee, the harness-attributed column is usually the one that earned trust — showing which failures were environmental, and which were the agent's own, made the rest of the numbers believable.

Run this checklist against your current setup

Ten questions. Any "no" is a gap worth an afternoon.

• Does every eval task run more than once, with the repeat count reported next to the figure?
• Do you report the worst run alongside the median, rather than the mean alone?
• Is a progress metric recorded per step, so every run can be plotted as a curve?
• Can you classify a run's curve shape without opening the raw trace?
• Do you log explicit signals for plan statement, plan changes and steps-since-progress?
• Do you count wasted steps — steps that produced no observable state change?
• Are harness events (truncation, timeout, rate limit, retry exhaustion, budget exhaustion) logged as a separate type from model decisions?
• Does every stored eval result carry both a model version and a harness version?
• Can you run the full eval with prior-task memory disabled, in one flag?
• Do you know your cost per eval cycle, measured from your own runs rather than estimated?

If you want a wider framing of reliability metrics that sit outside task success entirely, the taxonomy in Towards a Science of AI Agent Reliability (arXiv 2602.16666) is worth reading next; it proposes ways to evaluate agent reliability independently of whether the task was completed, which is the natural complement to everything above.

The uncomfortable conclusion

The finding in arXiv 2608.13417 that agents behave more like engineering optimisers than autonomous researchers is not a criticism of the agents. It is an accurate description of what they are currently good at, and it is genuinely useful — a competent optimiser that adapts and combines established techniques is worth a great deal, and most production work is exactly that. The problem is only that a final score cannot tell you whether you have one.

So measure the distribution rather than the point. Instrument framing, execution and feedback control, because they are cheap and they are where the fixes live. Plot the curve and learn its four shapes. Keep the harness in its own column. Treat memory as a feature under test rather than a gift. And when you report, report the worst run.

None of that requires a new benchmark or a new tool. It requires deciding that a long-horizon run is a trajectory rather than a verdict, and logging accordingly.