Prove You Can Build Evals: The Skill Gap That Gets You Hired

Almost everyone can call an API. Almost nobody can tell you if it got better

Sit on the other side of an interview loop for a few months and a pattern emerges that is almost boring in its consistency. Candidates can wire up a model. They can build retrieval. Many of them can put an agent loop together, handle tool calls, stream tokens to a front end and deploy the whole thing behind a gateway. Then you ask one question — how did you know the change you made was an improvement? — and the room goes quiet.

The honest answers are variations on the same theme. "It looked better." "The demo worked." "We tried a few prompts and picked the one we liked." Occasionally someone mentions a benchmark score from a model card, which is a measurement of somebody else's task, not theirs. Very rarely, someone says: I had a set of a few hundred real questions, I scored both versions on the same set, here is the per-item difference and here is why I believe it.

That last person gets hired. And the gap is not an accident of one company's hiring funnel — it is showing up in the market data too. Per one 2026 recruiting analysis from herohunt.ai, evaluation is now widely reported as the single biggest skill gap in AI engineering, with roughly 39.6 per cent of AI-first job postings listing evaluation-related skills as a requirement. Set that against the demand picture: one 2026 analysis from taggd.in reports AI engineer demand rising around 40 per cent year on year against a skilled talent pool growing only 15 to 20 per cent, with roughly 11.7 per cent of Indian job postings now explicitly requiring AI skills, up from 8.2 per cent a year earlier. Both figures come from single sources and should be read as directional rather than precise, but the direction is unambiguous, and it matches what the hiring side of the market sounds like from the inside. The same structural squeeze is visible in the India-UK AI hiring gap across job postings.

Why this gap is structural, not a temporary shortage

It would be comforting to think evaluation is under-supplied because it is hard. It is not especially hard. It is under-supplied because of three properties that make it get skipped everywhere, including inside well-run teams with budget and good engineers.

Evals are boring. Writing a runner that loops over three hundred questions and tallies scores is not the part of the job anyone joined for. It is data plumbing with a spreadsheet at the end. Nobody has ever been excited to build one, which means it falls to whoever cares enough to insist, and in most teams nobody insists.

Evals have no demo. This is the decisive one. Every other piece of AI work produces something you can show in a meeting — a chat window that answers, an agent that books something, a dashboard that lights up. An eval harness produces a table. You cannot put a table on a slide at a launch review and have anyone feel anything. In an organisation where visibility drives what gets resourced, an artefact with no demo loses every time.

Evals only pay off after launch. The value of a harness is entirely in the future: it is the thing that tells you, in month four, that the retrieval change you shipped quietly degraded answers for one class of question. Before launch it costs time and returns nothing you can point at. Teams under pressure to ship discount future value heavily, and the harness is the first thing cut.

Put those three together and you get a skill that is genuinely valuable, structurally neglected, and therefore rare — which is the exact shape of a hiring advantage. It is also, unusually, a skill you can prove from outside a company, because unlike a cost teardown or a production incident you do not need anyone's private data to build one.

What it is worth, with the usual caveats

Evaluation on its own is not a salary band, so treat what follows as context for the roles it opens rather than a price list for the skill. These are aggregator ranges as of 2026 and they vary enormously by city, sector, company stage and equity mix.

AI engineer compensation context as of 2026. Public salary-aggregator ranges, not guarantees.

Market and level	Reported range	Source and caveat
India, average base	Around ₹10 LPA, typical band roughly ₹6–16 LPA	Glassdoor, May 2026. Aggregated across cities and company types
India, entry level	Roughly ₹2–7 LPA	Wide because it mixes service firms, GCCs and product startups
India, 4–6 years	Roughly ₹10–15 LPA	GCC and product-company bands sit towards the top of this
India, senior AI architect or lead	Roughly ₹25–50+ LPA	The upper tail is heavily equity- and employer-dependent
UK, senior base	Roughly £90k–£150k	London-weighted; regional scale-ups typically sit below this band

One more market fact worth folding into how you build the artefact: more than 70 per cent of hiring managers prioritise familiarity with deployment tooling and cloud platforms such as AWS, Azure and GCP even for entry-level roles, per one 2026 portfolio guide. That is a single-source figure, but the implication is cheap to act on — a harness that runs in CI on a real cloud runner, against AWS Mumbai for an India-facing workload or London for a UK one, quietly satisfies that requirement while you are demonstrating something else entirely.

The artefact: one small, honest eval harness over a real task

Hiring managers consistently report wanting three to five production-ready projects with live demos and measurable impact rather than a list of tutorials. They also report, just as consistently, that generic chatbots, sentiment analysers and off-the-shelf fine-tunes are common to nearly every candidate and therefore differentiate nobody. An eval harness is the opposite: almost nobody has one, and the ones that exist are immediately legible as real work.

You are building one harness. Not a framework. Not a library with a plugin architecture. One harness over one task, done properly and published publicly. It contains exactly six things.

The six components of a portfolio eval harness, and what each one proves.

Component	What it is	What it signals to a reader
A versioned dataset file	JSONL of questions plus ground truth, committed to the repository with a content hash	You treat the measuring instrument as code, so results are reproducible
A runner	A loop that executes every variant over the same items in the same order and writes one results file per run	You can compare two things fairly, which is harder than it sounds
Checks, deterministic where possible	Programmatic assertions first; an LLM judge only for genuinely subjective dimensions	You know which measurements are free and exact and which are expensive and noisy
A results table across ≥2 variants	Two prompts, two models, or two retrieval configurations, scored side by side	You built it to answer a question, not to produce a number
A written interpretation	Two or three paragraphs saying what the numbers mean and what you decided	You can translate a measurement into a decision, which is the actual job
A regression gate	A CI step that fails the build when a defined metric drops below a threshold	You understand that an eval you do not run automatically is an eval you stop running

The single most important choice is the task, and the rule is short: pick something with a real failure mode. A harness over "does the model answer politely" proves nothing, because there is no interesting way to fail. A harness over "does the extractor pull the correct invoice total from a scanned purchase order, including when the document has two totals and one of them is a subtotal" proves a great deal, because the failure mode is specific, adversarial and checkable.

Good candidate tasks, all buildable from public data: extracting structured fields from filings or public tender documents; answering questions from a documentation corpus where some questions are deliberately unanswerable; classifying support tickets into a taxonomy with genuinely ambiguous boundary cases; choosing the right tool from a set of five where two of them overlap. If you are already working on agents, setting an agent reliability target and building the qualification harness for it is the same discipline applied to a harder surface.

Building the golden set: the part that decides everything else

The dataset is the harness. Everything downstream — the runner, the judge, the gate — is machinery that operates on it, and machinery cannot rescue a bad set. Get this right and the rest is an afternoon.

Source from real logs, not from memory. This is the trap that catches almost everybody. Sitting down to write a hundred test questions feels productive and produces a set that encodes your mental model of the task rather than the task. Real users ask questions with typos, with missing context, with two questions welded into one sentence, in a mix of English and another language, and about things the system was never designed to do. Nobody invents those at a desk. If you are building on an open-source application, run it, use it properly for a few days, and keep the transcript. If you have production access at work and permission to use it, that is better still.

De-identify before anything else touches it. Strip names, emails, phone numbers, account identifiers, addresses and anything that could re-identify a person, and do it as a scripted pass so it is repeatable rather than a one-off manual clean. This matters practically as well as legally: a golden set you cannot publish is a golden set that cannot be your portfolio piece. Replace rather than delete — swap a real customer name for a consistent pseudonym — so the linguistic shape of the question survives.

Stratify by intent class. Do not sample randomly and hope. Partition the traffic into classes that fail differently, then fill each class deliberately. For a document-QA system the classes might be: direct lookup, numeric extraction, multi-document comparison, out-of-scope, ambiguous phrasing, and adversarial or prompt-injection attempts. The out-of-scope and adversarial classes matter more than their share of real traffic suggests, because those are the classes where a change can quietly break something while the headline average stays flat.

Golden-set sizing guidance from 2026 practice, and what each tier buys you.

Set size	What it supports	What it cannot do
Below ~50	A smoke test. Useful for catching a total breakage	Per-run variance swamps the signal; you cannot distinguish improvement from noise
~100	A reasonable floor for one focused use case. Detects moderate regressions on the aggregate	Too thin per intent class to read class-level results with confidence
~300	Better. Enough per class to see where a change helped and where it hurt	Still not enough to detect a one-point move; nothing at this scale is
Synthetic top-up	Backfilling classes that are genuinely rare in the logs, such as adversarial inputs	Cannot be the foundation — synthetic questions inherit the generator's blind spots

Version it like code. The set lives in the repository, every change goes through a commit, and every results file records the content hash of the set it was run against. Without this you will eventually compare a run from March against a run from June, not notice that forty items changed in between, and draw a confident conclusion from an incoherent comparison. Refresh quarterly — retire items that no longer reflect the traffic, add items for failure modes you have since discovered — and record what you changed and why.

# golden_set.py - schema and fingerprint for the versioned evaluation set.
# Every row carries everything needed to score it, plus its provenance.

from dataclasses import dataclass
from typing import Literal, Optional
import hashlib
import json

IntentClass = Literal[
    "direct_lookup",
    "numeric_extraction",
    "multi_doc_compare",
    "out_of_scope",
    "ambiguous",
    "adversarial",
]


@dataclass(frozen=True)
class GoldenItem:
    id: str                          # stable, never reused, e.g. "gs-0147"
    intent: IntentClass              # stratification key
    question: str
    context_ids: tuple               # documents the runner is allowed to retrieve
    expected: dict                   # programmatic ground truth; {} if judged only
    rubric_id: Optional[str]         # judge rubric, only for subjective dimensions
    source: Literal["production_log", "synthetic"]
    added_on: str                    # ISO date - lets you age the set out
    notes: str = ""                  # why this item exists; future-you will need it


def fingerprint(items) -> str:
    """Content hash of the whole set. Gets written into every results file.

    If two runs disagree on this hash, they are not comparable. Full stop.
    """
    payload = json.dumps(
        [[i.id, i.question, i.expected] for i in sorted(items, key=lambda x: x.id)],
        sort_keys=True,
        ensure_ascii=False,
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]


def load(path: str):
    with open(path, encoding="utf-8") as fh:
        items = [GoldenItem(**json.loads(line)) for line in fh if line.strip()]

    ids = [i.id for i in items]
    assert len(ids) == len(set(ids)), "duplicate item ids in golden set"

    synthetic = sum(1 for i in items if i.source == "synthetic")
    if synthetic > len(items) * 0.35:
        # Synthetic items are a backstop, not a foundation. Over a third and
        # you are measuring the generator's imagination, not your users.
        print(f"WARNING: {synthetic}/{len(items)} items are synthetic")

    return items, fingerprint(items)

Choosing what to measure: deterministic first, judge last

The instinct, once you have an LLM to hand, is to ask it to score everything. Resist it. Every check you can make programmatically is free, instant, perfectly reproducible, and immune to a provider updating a model underneath you. Every check you delegate to a judge costs money, adds latency, introduces noise and creates a second system that itself needs validating.

The rule is: if a deterministic check can answer the question, a judge is the wrong tool. Reserve the judge for dimensions that are genuinely subjective — tone, helpfulness, faithfulness to a source passage, whether an explanation actually explains.

Task type mapped to the appropriate check type. Deterministic wherever the task allows.

Task type	Primary check	Check type	Judge needed?
Structured extraction (invoice, filing, form)	Schema validity plus exact or normalised match per field	Deterministic	No
Classification into a fixed taxonomy	Label match; per-class precision and recall	Deterministic	No
Tool or function selection	Tool-name correctness, then argument match	Deterministic	No
Constrained generation (JSON, SQL, a template)	Parses, validates against schema, executes without error	Deterministic	No
Numeric answer with tolerance	Absolute or relative tolerance band	Deterministic	No
Retrieval-grounded answer	Citation present and resolvable; claim supported by cited span	Hybrid	Partly — support is judged, citation presence is not
Refusal and safety behaviour	Refused or complied, against a labelled expectation	Deterministic on the binary, judged on the wording	Partly
Summarisation, explanation, tone	Rubric score against explicit criteria	Judge	Yes
Every task, always	Latency at p50 and p95, cost per task	Deterministic	No

That last row is not filler. Quality without cost and latency alongside it is half a measurement, and a harness that reports all three reads as production-minded rather than academic. If a variant wins on quality by two points and costs three times as much per task, the harness has not told you to ship it — it has told you to make a trade-off, and cost-aware evals that report quality per pound are how you present that honestly. The same reasoning drives routing decisions, where evaluating a model router by regret and escalation rate turns the trade-off into a metric in its own right.

Calibrating the judge, which is the step everyone skips

An uncalibrated LLM judge is a random number generator with a confident tone. The standard 2026 practice, consistent across the eval tooling ecosystem, is straightforward and almost nobody does it: calibrate the judge against a human-labelled gold set before you put it into production, measure the gap between judge verdicts and human labels, and iterate the judge prompt until that gap closes.

The loop in full:

One. Take a sample of your set — fifty to a hundred items is enough to start — and label it by hand against the rubric you are about to give the judge. Do this before you see any judge output, or you will anchor.

Two. Run the judge over the same items with the same rubric, and record its verdict and its stated reasoning.

Three. Compute agreement. Raw agreement is the headline; Cohen's kappa is worth reporting alongside it because it accounts for the agreement you would get by chance, which matters a lot when your labels are imbalanced.

Four. Read every disagreement. This is the step that actually produces the improvement. Each disagreement is a place where the rubric was ambiguous enough that two reasonable readers — you and the model — resolved it differently.

Five. Tighten the rubric into explicit, observable criteria and re-run. Repeat until agreement stops improving, then stop, and report the residual gap rather than hiding it.

A rubric, before and after tightening

Here is what step five actually looks like on a faithfulness dimension for a document-QA system. The before is the rubric almost everyone writes first.

A faithfulness rubric before and after calibration against human labels.

Version	Rubric text given to the judge	Observed problem
Before	"Rate the answer's faithfulness to the source document from 1 to 5, where 5 is fully faithful and 1 is not faithful at all."	Scores cluster on 4. The judge treats a confident tone as faithfulness and gives partial credit for plausibility. Two human labellers disagree with each other almost as much as they disagree with the judge, which is the real tell.
After	"Score 1 if the answer contains any claim not stated or directly entailed by the cited span. Score 0 otherwise. A claim is stated if it appears in the span; entailed if it follows from the span by arithmetic or by definition given in the span. Correct information that is absent from the span scores 1, because it is unsupported here regardless of truth. Omitting information scores 0 — incompleteness is scored separately. Output JSON: {"unsupported": 0 or 1, "offending_claim": string or null}."	Binary, observable, and it forces the judge to name the offending claim — which lets you spot-check its reasoning rather than trusting a bare number.

Three changes did the work. The scale collapsed from five points to a binary, because a five-point scale on a subjective dimension mostly measures the judge's mood. The criterion became observable rather than evaluative — "contains a claim not in the span" is checkable, "is faithful" is not. And the output became structured, with the judge required to cite the specific offending claim, which converts an opinion into something you can audit.

Four judge biases you must name in the write-up

Known LLM-judge failure modes as of 2026, and the mitigation for each.

Bias	What it does	Mitigation
Length bias	Longer answers score higher regardless of quality, because length correlates with apparent thoroughness	Report score against answer length; if they correlate, add an explicit instruction that length is not a criterion, and re-check
Position bias	In pairwise comparison, the option presented first (or last) wins disproportionately	Run every pair in both orders and average; report the disagreement rate between the two orderings as a diagnostic
Same-family bias	A judge favours outputs generated by models from its own family	Do not use a judge from the same family as any variant under test; if unavoidable, disclose it and cross-check a sample with a second family
Version drift	The provider updates the judge model and your measuring instrument silently changes between runs	Pin the exact judge model version in config, record it in every results file, and re-calibrate whenever you change it

The production pattern that falls out of all this is a split: use the LLM judge for the bulk run, because it is the only thing that scales to hundreds of items, and use sampled human review for the hardest cases and for periodic recalibration. Roughly ten per cent sampled review, weighted towards items where the judge expressed low confidence or where the two orderings disagreed, keeps the calibration honest without turning evaluation into a full-time labelling job.

Reading the results honestly, which is where most harnesses fail

You have a set, a runner and calibrated checks. Variant B scores 82 per cent and variant A scores 80 per cent on eighty examples. What do you say?

The answer that gets you hired is: nothing yet. A two-point move on eighty examples is roughly one and a half items. Run the same variant twice against a judge with any stochasticity and you will often see a move of that size with no change at all. Reporting it as a win is the single most common failure in candidate portfolios, and it is more damaging than having no eval, because it demonstrates that you will confidently mislead a team with a number.

Two techniques fix most of this.

Compare in pairs, not in aggregates. Run every variant over the same items in the same order, and compute the difference per item rather than the difference between two averages. Item difficulty varies enormously — some questions every variant gets right, some every variant gets wrong — and pairing cancels that variance out entirely. A paired comparison on a hundred items is often more sensitive than an unpaired comparison on four hundred.

Report a distribution, not a point. How many items improved, how many regressed, how many stayed the same, and how wide is the interval on the mean difference. Fourteen improvements against eleven regressions is a very different story from thirty improvements against one regression, even when both produce the same average.

# runner.py - executes every variant over the same items, then compares
# them in a paired fashion. Pairing is not an optimisation; it is the
# only way a set this small can detect a change at all.

import statistics


def run_variant(variant, items, checks, fingerprint):
    """One pass over the set. Deterministic order, one row per item."""
    rows = []
    for item in sorted(items, key=lambda i: i.id):
        out = variant.answer(item)
        rows.append({
            "item_id": item.id,
            "intent": item.intent,
            "variant": variant.name,
            "judge_model": variant.judge_model,   # pinned version, recorded
            "set_fingerprint": fingerprint,
            "score": checks.score(item, out),     # 0.0 - 1.0
            "latency_ms": out.latency_ms,
            "cost_usd": out.cost_usd,
            "raw": out.text,
        })
    return rows


def paired_delta(rows_a, rows_b):
    """Per-item difference between two variants over the same items."""
    a = {r["item_id"]: r["score"] for r in rows_a}
    b = {r["item_id"]: r["score"] for r in rows_b}

    shared = sorted(set(a) & set(b))
    if len(shared) != len(a) or len(shared) != len(b):
        raise ValueError("variants did not cover the same items - not comparable")

    diffs = [b[i] - a[i] for i in shared]
    n = len(diffs)
    mean = statistics.fmean(diffs)
    # Standard error of the mean difference. With n this small, the width
    # of the interval is usually the whole story.
    se = statistics.stdev(diffs) / (n ** 0.5) if n > 1 else float("inf")

    return {
        "n": n,
        "mean_delta": round(mean, 4),
        "std_error": round(se, 4),
        "ci95_low": round(mean - 1.96 * se, 4),
        "ci95_high": round(mean + 1.96 * se, 4),
        "improved": sum(1 for d in diffs if d > 0),
        "regressed": sum(1 for d in diffs if d < 0),
        "unchanged": sum(1 for d in diffs if d == 0),
    }


def verdict(result) -> str:
    """Plain language beats fake precision. Say what you can defend."""
    if result["ci95_low"] > 0:
        return "B is better; the interval excludes zero"
    if result["ci95_high"] < 0:
        return "B is worse; the interval excludes zero"
    return (
        "no detectable difference at n={n}; "
        "the interval spans zero, so this change is unproven"
    ).format(**result)

The agreement calculation for judge calibration is short enough to write out, and having it in the repository is disproportionately convincing because it shows the judge was treated as a system under test rather than as an oracle.

# agreement.py - judge versus human labels. Run this before you trust
# a single judged number, and re-run it whenever the judge model changes.

from collections import Counter


def agreement(human_labels: dict, judge_labels: dict):
    """Both dicts map item_id -> label. Labels must come from the same rubric."""
    shared = sorted(set(human_labels) & set(judge_labels))
    n = len(shared)
    if n == 0:
        raise ValueError("no overlapping items to compare")

    matches = sum(1 for i in shared if human_labels[i] == judge_labels[i])
    raw = matches / n

    # Cohen's kappa: agreement corrected for what chance alone would give.
    # Matters most when the labels are imbalanced, which they usually are.
    h_counts = Counter(human_labels[i] for i in shared)
    j_counts = Counter(judge_labels[i] for i in shared)
    labels = set(h_counts) | set(j_counts)
    expected = sum((h_counts[l] / n) * (j_counts[l] / n) for l in labels)
    kappa = (raw - expected) / (1 - expected) if expected < 1 else 1.0

    disagreements = [
        {"item_id": i, "human": human_labels[i], "judge": judge_labels[i]}
        for i in shared if human_labels[i] != judge_labels[i]
    ]

    return {
        "n": n,
        "raw_agreement": round(raw, 4),
        "cohens_kappa": round(kappa, 4),
        # Read every one of these. They are the rubric's ambiguities,
        # written out for you, and fixing them is the whole calibration loop.
        "disagreements": disagreements,
    }

When you write results up, say what you can defend in plain language. "Variant B improved twenty-nine items and regressed four; the mean difference was 0.09 with a 95 per cent interval from 0.04 to 0.14, so I shipped it" is a defensible sentence. "B achieved 91.3 per cent versus 89.7 per cent" is three significant figures of noise.

The write-up: ninety seconds from opening it to understanding it

A hiring manager opening your repository gives it somewhere between sixty and a hundred and twenty seconds before deciding whether to read properly. The README is the artefact; the code is the evidence behind it.

The framing rule matters more than the format. Outcome framing beats metric framing: "reduced manual review time by 40 per cent" lands harder than "achieved 91 per cent F1 on the test set", because the first names something a business cares about and the second names something only you care about. Treat that as a principle for how you phrase whatever your harness actually found, not as a number to claim.

Three things, in this order, in the first screen:

The decision it enabled. Not what you built — what you were able to decide because you built it. "This harness is how I decided not to upgrade the retrieval pipeline." That sentence tells a reader immediately that the eval existed to serve a decision.

The number that moved, with its uncertainty. One number, its interval, and the sample size. Not a table of eleven metrics.

The trade-off you accepted. Every real result has one. Faster but less accurate on one class. More accurate but forty per cent more expensive per task. Naming it is what separates an engineer from a demo.

# README.md - the template. Fill each line; delete nothing.

## What this measures
<One sentence: the task, and the specific failure mode it targets.>

## The decision it enabled
<One sentence: what you did differently because of the result.>

## Result
<Variant A> vs <Variant B>, n=<items>, paired.
Mean difference: <x> (95% CI <low> to <high>). Improved <i>, regressed <r>.
Cost per task: <a> vs <b>. Latency p95: <a> vs <b>.

## The trade-off I accepted
<One sentence. There is always one.>

## How the set was built
<Source, size, intent-class breakdown, synthetic share, date last refreshed.>

## How the judge was calibrated
<Sample size hand-labelled, raw agreement, kappa, judge model version pinned.>

## What this eval does NOT catch
<The blind spot. Be specific. This section is why people trust the rest.>

The last section is the one that converts. Every experienced reader knows a harness has blind spots, so naming yours reads as competence rather than weakness, and a write-up without it reads as marketing. The broader mechanics of getting an AI project read in one pass are covered in writing up an AI project so it survives a ninety-second scan, and the same discipline applied to a cost story is in proving you can cut a token bill.

Here is the part that is easy to get wrong: a harness nobody can find is not proof of work. It is a private file. A repository URL buried in the third page of a CV, or a post that scrolled past four months ago, does not function as evidence, because the people doing the finding — a hiring manager at an Indian GCC in Bengaluru, a founder at a Manchester scale-up — are scanning profiles, not excavating archives. A Verified Builder profile is where the harness, its write-up and the rest of your projects sit on one page, with your location, availability and the rest of your work attached, in a format built to be scanned rather than reconstructed. That is the whole argument for having one, and it takes about two minutes.

The interview: eight questions you will be asked about your harness

Once the harness is public, the conversation changes shape. Instead of being asked to describe hypothetical approaches, you get asked about a real thing you built, which is an enormously better position. These are the questions that come up, and what a strong answer contains.

1. "How do you know your judge is right?" The answer is a number and a method, never a reassurance. Name the size of the hand-labelled sample, the raw agreement, the kappa, and the fact that the judge model version is pinned. Then say where you still disagree with it and why you accepted that residual.

2. "What would you do if the golden set leaked into training data?" The honest answer starts by conceding that for any public dataset this has probably already happened. Then the mitigations: hold back a private slice that has never been published; include items constructed from data postdating the model's cutoff; watch for the signature of contamination, which is a model performing far better on your set than on fresh items of the same class; and rotate the set on a schedule rather than treating it as permanent.

3. "How do you know a two-point improvement is real?" Paired comparison, the distribution of per-item differences, and the interval. If the interval spans zero, the change is unproven and you say so. Bonus points for volunteering the sample size at which the move would have been detectable.

4. "Why did you use a judge here and not there?" The deterministic-first principle. Walk through one dimension you check programmatically and one you judge, and explain why the first could not be judged usefully and the second could not be checked programmatically.

5. "How does this run in CI, and what does it block?" Describe the gate: which metric, which threshold, whether it blocks the merge or only warns, and how you handle judge cost and flakiness on every pull request. The realistic answer usually involves a fast deterministic subset on every commit and the full judged run nightly or on release branches.

6. "What did you find that surprised you?" Every real harness produces at least one surprise, usually a class of question that was failing quietly. If you have no answer here, the interviewer concludes you ran it twice and stopped.

7. "What would you change if you had another week?" Tests whether you know the harness's weaknesses or are attached to it. Have two specific answers ready — more items in a thin intent class, a second judge family for cross-checking, latency measured under concurrency rather than serially.

8. "What did this eval fail to catch?" This is the one that matters most, and it is often the last question asked. Being able to name your own harness's blind spot — "it scores single-turn answers, so it tells me nothing about whether the assistant degrades over a ten-turn conversation", or "every item is in English, and roughly a third of our real traffic is not" — is the strongest signal available to you in the entire loop. It demonstrates that you understand evaluation as a bounded instrument rather than a source of truth, which is precisely the judgement that makes someone safe to put in charge of a quality bar. A candidate who claims their harness catches everything has told the interviewer they have not looked hard enough at it.

If part of the loop involves demonstrating the system live rather than discussing it, the mechanics of demoing an AI agent live in an interview apply on top of all this — the harness gives you something to fall back on when the demo misbehaves, which it will.

Five pitfalls that sink an otherwise good harness

Judging on the examples you tuned on. Covered above, and worth repeating because it is the most common and the most quietly fatal. Split the set on day one, before you have any results you are attached to.

An unpinned judge model. If your configuration names a model alias that resolves to whatever the provider ships this month, your measuring instrument changes underneath you and every historical comparison becomes meaningless. Pin the version, record it in the results file, and re-calibrate deliberately when you change it.

Refusing to ship a negative result. The instinct is to keep iterating until the number goes up so the write-up has a win in it. This is backwards. A published null result with a proper interval is a stronger hiring signal than a marginal win, because anyone can produce a marginal win by looking at enough variants. Report what you measured.

Building the harness for a task nobody cares about. A technically flawless evaluation of a solved benchmark reads as an exercise. Pick a task where the failure mode is real and the answer is genuinely unknown before you run it.

Over-engineering into a framework. The temptation, about four days in, is to abstract the runner into a plugin system with adapters and a configuration DSL. Do not. Nobody is going to adopt your eval framework, and a reader trying to understand a plugin architecture in ninety seconds will close the tab. One task, three hundred items, a runner that fits on two screens, a results table and a written interpretation. The constraint is the point.

None of this requires permission, a job, or anyone's private data. Pick a task with a real failure mode, gather a few hundred honest questions, measure two variants properly, and write down what you found — including the part you could not measure. As of 2026, that puts you in a small minority of people who can answer the question that ends most interviews, and it is a fortnight's work.