What you need to know

  • A leaderboard and a procurement decision are different questions. Public benchmarks rank agents on somebody else's distribution. You are buying against yours.
  • Your git history is already a benchmark. Every merged bug-fix pull request that changed source and touched tests is a candidate task with a built-in grader.
  • Validation is the whole game. If the new tests do not fail at the parent commit and pass at the merge commit, it is not a task. Discard it.
  • Grade by execution, never by resemblance. Fail-to-pass tests prove the fix; pass-to-pass tests prove nothing else broke. Both must hold.
  • An undisclosed harness makes a number meaningless. Record model, scaffold, tool set, turn cap, timeout and container digest as a run manifest.
  • One run per task is noise. Sample each task several times, report pass@1 with an interval, and be honest that a small task count means a wide one.
Pro tip

Write down the decision the benchmark is supposed to inform before you mine a single commit — "which agent do we give to the platform team for a twelve-month licence" is a very different target from "can an agent close our stale-bug backlog unattended". The first needs breadth across your task mix; the second needs depth on one category. Building the wrong harness well is still building the wrong harness.

What SWE-bench answers, and what it does not

SWE-bench is a large-scale, execution-based benchmark for end-to-end software engineering by LLM-based agents: 2,294 task instances collected from real GitHub issues and their corresponding pull requests across twelve widely used open-source Python repositories, with agents producing code edits represented as patch files. That design was a genuine advance. Grading by running tests rather than by string similarity to a reference solution is what made agentic coding measurable at all, and the same principle is the backbone of the private harness described below.

The problem is not that SWE-bench is bad. It is that a strong score certifies competence on a specific distribution, and your repository is not that distribution. Three documented limitations each bite directly on a procurement decision. Overly detailed issue descriptions inflate resolution rates — many benchmark issues effectively specify the fix, so the measured skill is closer to instruction-following than to diagnosis, and your tickets are rarely that generous. The single-language bias towards Python limits generalisability; if your production system is Kotlin, Go or a TypeScript monorepo, the transfer is an assumption, not a finding. And scaffold and model effects are confounded, so a leaderboard row cannot tell you whether the result came from the model, the harness around it, or the interaction of the two.

The field has answered with a wave of complementary benchmarks rather than a single replacement, which is itself informative. As of July 2026 the landscape looks roughly like this.

Benchmark What it measures What it does not tell you
SWE-bench End-to-end issue resolution on real GitHub issues and pull requests across twelve open-source Python repositories, graded by execution. Behaviour outside Python or on terse tickets; whether model or scaffold earned the score.
SWE-bench Pro Harder, multi-file, contamination-resistant tasks. Your repository's idioms, build system or test topology.
SWE-Compass Evaluation extended across eight programming languages and multiple task types. Depth within any one stack; your team's actual mix of task types.
FeatureBench Agentic coding on complex feature development rather than bug repair. Maintenance, upgrade and regression work.
SWE-Chain Chained release-level package upgrades — the sequential dependency work most benchmarks skip. Greenfield feature quality or single-issue repair.
UTBoost Rigorous evaluation of coding agents on SWE-bench, tightening how results are established. Anything about non-SWE-bench distributions.
Terminal-Bench Shell and command-line competence — the environment work agents do between edits. Code quality or long-horizon design.
LiveCodeBench Code generation on public programming problems. Repository-scale navigation and multi-file editing.

The reporting guidance emerging from this literature is worth adopting even internally: lead with SWE-bench Verified for engineering, LiveCodeBench for generation and Terminal-Bench for shell; quote at least two together rather than cherry-picking one; and disclose the harness. "Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering" (arXiv 2606.17799) argues the misalignment is structural, and "Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures" (arXiv 2604.03515) shows how much variation hides behind the word "agent". None of that makes public benchmarks useless — it makes them a screening filter for shortlisting two or three agents worth testing properly on your own code.

Watch out

Treating a leaderboard position as a purchasing decision is the failure mode this guide exists to prevent. A tool that resolves a high share of Python issues written by open-source maintainers may still flounder on a service whose test suite takes nine minutes to boot, whose conventions live in an internal wiki, and whose tickets say "checkout broken on staging, see thread". You will not discover that from a public number. You will discover it in month three of a licence.

What you need before you start

A private harness is only as good as the repository underneath it. You need a real test suite — one where a meaningful share of behaviour is covered and tests fail for behavioural rather than environmental reasons. You need reproducible CI: a documented, deterministic way to install dependencies and run tests that does not depend on a developer's laptop. You need at least a few dozen merged bug-fix pull requests with linked issues, because that is your raw ore; a repository three months old has nothing to mine.

You also need container-based isolation. Every task must run in a fresh container built from a pinned image, because an agent that modifies global state, installs a package or leaves a stray file will otherwise contaminate every subsequent task in the sweep. And you need a budget: a hundred tasks, four samples each, three candidate agents is twelve hundred agent runs, each involving dozens of model turns and a full test-suite execution. Price that before you begin rather than halfway through.

Mining tasks from your own git history

This is the core technique, and it is more mechanical than it sounds. A good candidate is a merged pull request that changes source files and adds or modifies test files, has a linked issue giving you natural-language problem text, and is small enough that a single agent session could plausibly complete it. The pairing matters: the test delta is the grader, and without it you have no objective way to say whether an agent succeeded.

Pull candidates out of git and filter aggressively. The sketch below is deliberately framework-agnostic — it shells out to git and assumes nothing about your forge beyond being able to fetch issue text separately.

import re
import subprocess

SOURCE_RE = re.compile(r"^(src|lib|app|internal)/.*\.(py|ts|go|kt|java)$")
TEST_RE = re.compile(r"(^|/)(tests?|spec)/|(_test|\.test|_spec)\.")
ISSUE_RE = re.compile(r"(?:closes|fixes|resolves)\s+#(\d+)", re.IGNORECASE)


def run(*args):
    return subprocess.run(args, capture_output=True, text=True, check=True).stdout


def merged_commits(since="2 years ago", limit=2000):
    """Merge commits are the unit of work: one PR, one candidate task."""
    log = run("git", "log", "--merges", "--since", since,
              f"--max-count={limit}", "--format=%H%x00%P%x00%s%x00%b%x1e")
    for record in log.split("\x1e"):
        if not record.strip():
            continue
        sha, parents, subject, body = record.strip().split("\x00")
        yield sha, parents.split(), f"{subject}\n{body}"


def changed_files(base_sha, head_sha):
    out = run("git", "diff", "--name-only", base_sha, head_sha)
    return [line for line in out.splitlines() if line]


def mine_candidates():
    candidates = []
    for sha, parents, message in merged_commits():
        if len(parents) != 2:
            continue                       # not a standard two-parent merge
        base = parents[0]                  # mainline parent = pre-change state
        files = changed_files(base, sha)
        source = [f for f in files if SOURCE_RE.match(f)]
        tests = [f for f in files if TEST_RE.search(f)]
        issue = ISSUE_RE.search(message)

        # Reject: no test delta, no source delta, no linked issue, or too large.
        if not source or not tests or not issue:
            continue
        if len(files) > 25:
            continue                       # sprawling refactors are not tasks

        candidates.append({
            "task_id": f"{sha[:12]}",
            "base_commit": base,
            "merge_commit": sha,
            "issue_number": int(issue.group(1)),
            "source_files": source,
            # File paths double as selectors for most runners; convert here if yours differs.
            "test_selectors": tests,
        })
    return candidates

Expect brutal attrition. On a healthy repository with a couple of years of history, it is normal for a few hundred merge commits to yield a few dozen candidates and for only a fraction of those to survive the validation step that follows. That is fine. A small set of tasks you trust completely beats a large set you half-believe, and the discarded ones cost you nothing but compute.

Validating a candidate: fail at parent, pass at merge

A candidate is not a task until it has been proven to behave the way a task must. The validation loop is simple and non-negotiable: check out the parent commit, apply only the test changes from the merge, run those tests, and assert that they fail. Then check out the merge commit, run the same tests, and assert that they pass. If either assertion does not hold, the candidate is silently broken — an environment-dependent test, a fix that landed in an earlier commit, a test that does not exercise the changed behaviour — and it must be discarded.

import json
import os
import subprocess

TIMEOUT_S = 900


def run_tests_in_container(image, commit, test_selectors, overlay_patch=None):
    """Run a test selection at a specific commit inside a pinned image.
    Returns {test_id: 'passed' | 'failed' | 'error'} parsed from a report."""
    script = [
        f"git checkout --quiet --force {commit}",
        "git clean -xfd --quiet",
    ]
    if overlay_patch:
        script.append(f"git apply {overlay_patch}")   # test-only changes
    script.append(
        "run-tests --report /out/report.json " + " ".join(test_selectors)
    )
    out_dir = os.path.abspath("./out")   # bind sources must be absolute
    try:
        subprocess.run(
            ["container", "run", "--rm", "--network=none",
             "--mount", f"type=bind,src={out_dir},dst=/out",
             image, "bash", "-lc", " && ".join(script)],
            timeout=TIMEOUT_S, check=False,
        )
    except subprocess.TimeoutExpired:
        # A timeout is a failure, not a crash — score it and move on.
        return {"status": "timeout", "fail_to_pass": [], "pass_to_pass": []}
    with open(os.path.join(out_dir, "report.json")) as fh:
        return json.load(fh)


def validate(candidate, image, test_only_patch):
    """A task exists only if its tests fail before the fix and pass after it."""
    selectors = candidate["test_selectors"]

    before = run_tests_in_container(
        image, candidate["base_commit"], selectors, overlay_patch=test_only_patch
    )
    fail_to_pass = [t for t, status in before.items() if status == "failed"]
    if not fail_to_pass:
        return None            # nothing failed pre-fix: not a task

    after = run_tests_in_container(image, candidate["merge_commit"], selectors)
    if any(after.get(t) != "passed" for t in fail_to_pass):
        return None            # did not flip cleanly: not a task

    # Everything green at BOTH commits is the regression guard.
    pass_to_pass = sorted(
        t for t, status in after.items()
        if status == "passed" and before.get(t) == "passed"
    )
    candidate["fail_to_pass"] = sorted(fail_to_pass)
    candidate["pass_to_pass"] = pass_to_pass
    return candidate

Run the validation twice on different days before you trust a task. A test that fails at the parent commit for a reason unrelated to the bug — a clock, a locale, a leftover cache — will look perfectly valid on the first pass and quietly poison every comparison afterwards.

Recommended

Store the validated task set in version control as plain JSON, one record per task, with the base commit, merge commit, test selectors, fail-to-pass list and pass-to-pass list. Treat it as a dataset with its own change history. When you add tasks six months later you want to be able to say precisely which subset a past result was measured on, otherwise every historical number becomes uncomparable the moment the set grows.

Building the task instance: what the agent sees

The task instance is what you hand the agent, and the discipline is entirely about what you leave out. The agent gets the natural-language issue text, the repository checked out at the parent commit, and the command it may use to run tests. It does not get the tests it will be graded on, and it does not get the real patch. Both are grading artefacts; leaking either turns the benchmark into a memorisation exercise that flatters every agent equally and discriminates between none.

Withholding the grading tests is fiddlier than it sounds, because the tests usually live in the repository the agent is exploring. The clean approach is to grade against the merge commit's test files while giving the agent the parent commit's tree, applying the test-only overlay after the agent has finished and its patch has been captured. If the fail-to-pass tests exist in some form at the parent commit, quarantine them from the agent's working tree and restore them at grading time.

The finding that overly detailed issue descriptions inflate resolution rates deserves a direct response, because it is the single largest source of optimism in public numbers. Control for it by running each task twice with different problem statements: once with the original issue text as written, and once with a deliberately terse restatement of a sentence or two that names the symptom and nothing else. "Checkout total is wrong when a discount code is applied to a multi-currency basket" is a terse restatement. "In pricing/discount.py, the rounding in apply_discount() occurs before currency conversion" is not — it is a fix in disguise. The gap between the two conditions is one of the most useful numbers the exercise produces, because your real tickets look far more like the terse version. This is where our guidance on building evals from production logs pays off: the vocabulary of your actual bug reports is already sitting in your issue tracker.

Grading: fail-to-pass, pass-to-pass and the regression guard

Grading is execution-based only. The agent's output is a patch; you apply it to the repository at the parent commit, restore the grading tests, and run them. There is no partial credit for a plausible-looking diff, no similarity score against the human patch, and no LLM judge in the loop. The question is binary and the machine answers it.

A task counts as resolved when two conditions hold together. Every fail-to-pass test — the ones that failed at the parent and passed after the real fix — must now pass. And every pass-to-pass test must still pass. The second condition is the one teams skip and the one that matters most. An agent that repairs the reported bug while breaking four unrelated tests has not fixed the bug; it has moved it. That behaviour is common enough with aggressive agents that a harness without a regression guard will systematically overrate exactly the agents you least want to deploy unattended.

Keep the pass-to-pass set as broad as your runtime budget allows. Restricting it to the same module is cheap and catches almost nothing; the full suite is expensive and catches everything. A reasonable compromise is the module's full suite plus any test file that imports the changed source files, with a periodic full-suite sweep on a subsample to confirm the narrower guard is not missing damage. This is a different discipline from asking a model to critique a diff — for that layer, our guide to AI code review as a CI quality gate is the companion piece.

Avoid

Do not grade on lint, type-check or formatter output. They are cheap to satisfy, they correlate weakly with whether the software works, and an agent optimising against them will produce beautifully formatted code that does not fix anything. Style checks belong in your CI pipeline, not in your benchmark's scoring function.

Disclose the harness or the number means nothing

Two teams can run the same model on the same tasks and get materially different results, because different harness implementations can significantly affect benchmark results. That is the argument in "Stop Comparing LLM Agents Without Disclosing the Harness" (arXiv 2605.23950), and it applies to your internal comparison exactly as it applies to a public leaderboard. Swap an agent's turn cap from twenty to sixty between runs and you have not measured two agents; you have measured a configuration change and an agent change, entangled beyond separation.

The fix is a run manifest emitted with every sweep and stored beside the results. Nothing in it is exotic; the point is that it is recorded automatically rather than remembered approximately.

Manifest field Why it changes the result
model and model_versionThe obvious variable — and the one most often logged as a family name rather than a pinned version.
scaffold and scaffold_versionPlanning loop, file navigation and edit strategy live here; model and scaffold effects stay confounded unless both are pinned.
toolsAn agent with a test-runner tool behaves nothing like one restricted to file edits.
max_turnsBounds how many diagnose-edit-test cycles are possible.
context_limitSets how much of the repository can be held before truncation.
retry_policySilent retries on tool errors rescue runs that would otherwise fail.
temperature / samplingSets run-to-run variance — the thing your statistics are trying to measure.
task_timeout_sA timeout is a scored failure; changing it changes the score.
image_digestA dependency bump inside the container can flip tests with no agent involvement.
 
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "coding-agent-run-manifest",
  "type": "object",
  "required": [
    "run_id", "created_at", "task_set", "agent", "environment"
  ],
  "properties": {
    "run_id":     { "type": "string" },
    "created_at": { "type": "string", "format": "date-time" },
    "task_set": {
      "type": "object",
      "required": ["name", "revision", "task_count"],
      "properties": {
        "name":        { "type": "string" },
        "revision":    { "type": "string", "description": "git sha of the task-set file" },
        "task_count":  { "type": "integer", "minimum": 1 },
        "prompt_mode": { "enum": ["original_issue", "terse_restatement"] }
      }
    },
    "agent": {
      "type": "object",
      "required": ["model", "model_version", "scaffold", "scaffold_version"],
      "properties": {
        "model":            { "type": "string" },
        "model_version":    { "type": "string" },
        "scaffold":         { "type": "string" },
        "scaffold_version": { "type": "string" },
        "tools":            { "type": "array", "items": { "type": "string" } },
        "max_turns":        { "type": "integer" },
        "context_limit":    { "type": "integer" },
        "temperature":      { "type": "number" },
        "retry_policy":     { "type": "string" }
      }
    },
    "environment": {
      "type": "object",
      "required": ["image_digest", "task_timeout_s"],
      "properties": {
        "image_digest":   { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
        "task_timeout_s": { "type": "integer" },
        "network":        { "enum": ["none", "proxy", "open"] },
        "cpu_limit":      { "type": "string" },
        "memory_limit":   { "type": "string" }
      }
    },
    "samples_per_task": { "type": "integer", "minimum": 1 }
  }
}

The operational rule that follows is short: if two manifests differ in any field, the two result sets are not comparable, and any table that places them side by side needs a footnote saying so.

Every article here is written by a Verified Builder. Want your name on the next one?

AI Tech Connect lists AI engineers, founders and researchers across India and the UK — and the people hiring browse it to find them. Adding your profile is free.

Become a Verified Builder →

Contamination control, and its honest limits

The strongest argument for a private harness is contamination resistance. Public benchmarks face a structural problem: their tasks come from public repositories, so the issues, the discussions and often the exact fixing patches can end up in training corpora. When that happens the benchmark stops measuring problem-solving and starts measuring recall, and it does so invisibly — the agent still produces a correct patch, just not for the reason you assume. This is precisely why SWE-bench Pro was built to be contamination-resistant and why Tasks mined from a private repository sidestep the issue almost entirely: nobody trained on your internal payments service, so the agent has to navigate and reason rather than recall.

Be honest about the caveats, though. If your repository is public — many companies open-source substantial parts of their stack — your mined tasks carry the same contamination risk as any public benchmark, and possibly more, because they are less likely to have been deliberately filtered. Two mitigations help: prefer commits merged after a model's known training cutoff, accepting that cutoffs are approximate and self-reported, and prefer genuinely internal repositories where you have them. Whatever you do, write the choice into the results summary. "37 tasks, 29 from a private service, 8 from a public repository, all merged after Q1 2026" lets a reader discount your numbers appropriately. A percentage with no provenance does not.

How many tasks, how many samples

This is where most internal benchmarks quietly fall apart. A team runs each agent once over fifteen tasks, sees eleven versus nine, and declares a winner. That difference sits well inside the noise of a coin flip, and the decision it justifies costs six figures.

Two things fix it. The first is task count: thirty to fifty validated tasks is a workable minimum for a real procurement decision and a hundred is materially better, because below thirty the interval around a pass rate is so wide that almost no realistic difference will clear it. The second is repeated sampling. Coding agents are stochastic — the same agent on the same task can succeed and fail on consecutive attempts — so run k independent samples per task, typically three to five, and report both the mean success rate and its stability.

import math
from collections import defaultdict


def pass_at_1(results):
    """results: list of {'task_id': str, 'resolved': bool}, k samples per task.
    pass@1 = mean per-task success probability, estimated from k samples."""
    per_task = defaultdict(list)
    for row in results:
        per_task[row["task_id"]].append(bool(row["resolved"]))

    rates = [sum(runs) / len(runs) for runs in per_task.values()]
    n = len(rates)
    mean = sum(rates) / n

    # Per-task variance, then the standard error of the mean across tasks.
    var = sum((r - mean) ** 2 for r in rates) / (n - 1) if n > 1 else 0.0
    stderr = math.sqrt(var / n)
    half_width = 1.96 * stderr          # ~95% normal-approximation interval

    always = sum(1 for r in rates if r == 1.0)     # solved on every sample
    never = sum(1 for r in rates if r == 0.0)      # solved on no sample
    return {
        "tasks": n,
        "pass@1": round(mean, 4),
        "ci95": (round(max(0.0, mean - half_width), 4),
                 round(min(1.0, mean + half_width), 4)),
        "stable_solved": always,        # pass^k: reliable wins
        "never_solved": never,
        "flaky": n - always - never,    # sometimes yes, sometimes no
    }

Always report the interval, and treat overlapping intervals as "no measured difference" rather than a narrow win. The stable_solved figure — tasks solved on every sample, a pass^k style measure — is often more decision-relevant than the mean: a task an agent solves four times out of four is work you can delegate, while a task it solves twice out of four is work you must supervise, so a lower pass@1 with a higher stable-solved count may be the better operational choice. If you also want to measure how the agent got there rather than only whether it arrived, our guide to evaluating agents on trajectory, tool calls and outcome covers the process side that execution grading deliberately ignores.

Cost and runtime control

A full sweep is embarrassingly parallel — tasks are independent, so the only real limits are API rate limits and CI concurrency. Four levers keep the bill sane. Cache the container base image and bake dependencies into it, since installation repeated across a thousand runs dwarfs the actual test time. Cap turns explicitly and treat the cap as a manifest field, because uncapped agents burn tokens on tasks they were never going to solve. Set a per-task timeout and score timeouts as failures, which they are. And run a cheap single-sample pilot over five tasks first, so harness bugs surface before you spend the full budget.

Regional economics change the answer, not just the number. Agent API pricing is broadly uniform, but the CI runners executing the tests are not: compute costs, egress charges and the exchange rate all move between an India-hosted runner pool and a UK or EU one, so a team in Bengaluru and a team in London running the identical sweep will see meaningfully different totals. If your evaluation crosses regions, price it in local currency and record the region in the manifest, or you will spend a meeting reconciling two "same" figures that were never the same.

Reading the results

The most common outcome of a well-built private harness is not a clean winner. It is a split: agent A leads on single-file bug repair while agent B leads on multi-file feature work, or one agent is stronger on the original issue text while another holds up better under terse restatements. That split is the finding, not a failure to produce one.

Act on it by weighting task categories to your actual workload mix. If your engineering time is seventy per cent maintenance and thirty per cent new features, an unweighted average answers a question nobody asked. Tag every mined task with a category during validation, report per-category rates, and weight the summary by the mix you observe in your own tracker.

Illustrative example only — fictional numbers from a fictional repository, not measurements of any real product
Task category Tasks Agent A pass@1 Agent B pass@1 Workload weight
Single-file bug repair180.61 (0.44–0.78)0.50 (0.33–0.67)45%
Multi-file feature work120.29 (0.13–0.45)0.42 (0.24–0.60)30%
Dependency / upgrade chores90.44 (0.23–0.65)0.44 (0.23–0.65)15%
Test-only additions60.67 (0.42–0.92)0.58 (0.32–0.84)10%
Weighted total450.490.47—

Read that table the way you should read your own: the weighted totals are statistically indistinguishable, the per-category picture is where the decision lives, and every interval is wide because forty-five tasks is a small sample. A table like this argues for splitting work between two agents far more persuasively than for a single licence — and if that is where your data lands, say so.

From a verified Builder

"We spent two weeks arguing about leaderboard rows and one weekend mining forty tasks from our own history. The private harness reversed the ranking we had assumed — the agent we were about to standardise on was strong on the tickets our tech leads wrote and noticeably weaker on the ones that came in from support, which is most of them. The whole exercise cost less than a fortnight of the licence we nearly signed."

— Aditi, Verified Builder · Bengaluru, India

Pitfalls to avoid

  1. Flaky tests in the graded set. A test that fails intermittently will attribute randomness to the agent. Run every candidate's validation at least twice on different days and quarantine anything that wavers.
  2. Network-dependent tests. A task that hits an external API measures that API's uptime as much as the agent. Run containers with networking disabled and drop tasks that cannot pass without it.
  3. Issue text that contains the fix. Tickets written by the engineer who already diagnosed the problem inflate results exactly as detailed public issue descriptions do. Grade on a terse restatement alongside the original.
  4. Grading on lint or formatting. Cheap to satisfy, weakly correlated with working software, and a direct invitation to optimise the wrong thing.
  5. Letting the agent see the grading tests. If the fail-to-pass tests are visible in the working tree, you are measuring reading comprehension. Quarantine them and restore at grading time.
  6. Comparing across different harnesses. Different turn caps, tool sets or container digests make two runs incomparable. If the manifests differ, the numbers do not belong in the same table.
  7. Declaring a winner from one run over fifteen tasks. Single-run comparisons on small sets are noise wearing a decision's clothing. Sample repeatedly, report intervals, and accept "no measured difference" as a legitimate result.

Where this leaves you

Public coding benchmarks are a screening tool and a research instrument, and they are good at both. They are not a procurement decision, because they cannot be — they measure a distribution that is not yours, through a harness that is not yours, on tasks that may already sit in a training corpus. Use them to shortlist two or three agents, then answer the question you actually care about with your own code. If you are still narrowing that shortlist, our comparison of Claude Code, Cursor and Codex is the screening step that comes first.

The build order is a weekend of code and a week of elapsed validation. Mine merge commits that changed source and tests and carry a linked issue. Validate each by proving its tests fail at the parent and pass at the merge, and discard everything that does not. Assemble task instances that withhold the grading tests and the reference patch, with a terse restatement of each issue alongside the original. Grade by execution on fail-to-pass and pass-to-pass together. Emit a run manifest so your numbers stay comparable over time. Sample each task several times, report intervals, and weight the categories by your real workload mix. Then rerun it next quarter, when the model names have changed and the method has not — and once it is stable, wire a subset into your pipeline, which our guide to putting evals in CI covers.

One last thing. A sanitised version of the harness — the mining and validation code, the manifest schema, the aggregation logic, with your proprietary tasks stripped out — is unusually strong proof of work for an AI engineer, because it demonstrates evaluation judgement, reproducibility discipline and statistical honesty in a single artefact. Our guide to shipping a public agent and eval harness as proof of work takes that further.