Building a human annotation pipeline for LLM evals: golden sets that hold up

What you will build

Every automated evaluation you run — an LLM-as-a-judge scoring answers, a regression suite gating your deploys, a dashboard tracking quality over time — ultimately rests on a set of human decisions about what "good" looks like. Those decisions live in a golden set: a curated collection of inputs paired with human-verified labels or reference outputs. Get the golden set right and everything downstream becomes trustworthy. Get it wrong and you are optimising confidently towards the wrong target.

This guide walks through building the pipeline that produces and maintains that golden set: how to design the labelling task, how to measure whether your annotators actually agree, which self-hostable tools to use, how to feed the pipeline from production traffic, and how to keep the whole thing compliant with India's DPDP Act and the UK GDPR. It is written for small builder teams — a two-person start-up in Bengaluru, a lean data team in Manchester — not for organisations with a hundred-person labelling operation. The principles scale down cleanly.

By the end you should be able to stand up a working annotation loop, report an inter-annotator agreement number you can defend, and calibrate an automated judge against real human ground truth. If you have already read our companion pieces on the evaluation suite that pairs golden sets with judges and on rubric design and judge calibration, this is the layer beneath both of them.

Why judges need human ground truth

The appeal of an LLM-as-a-judge is obvious: it is fast, cheap relative to human review, and it scales to thousands of examples overnight. The trap is equally obvious once you name it. A judge is a model, and models have systematic biases. Left unchecked, an LLM judge tends to reward longer answers over shorter correct ones (verbosity bias), to prefer whichever response happens to appear first or last (position bias), and — most awkwardly — to rate outputs from its own model family more highly than it should (self-enhancement bias).

None of these biases announce themselves. A judge can be wrong in a perfectly confident, well-argued paragraph. The only way to know whether your judge is measuring quality or measuring its own preferences is to compare its verdicts against a set of labels you trust independently — human labels. That comparison is calibration, and it is not optional. A judge you have not calibrated against human ground truth is a number generator, not an evaluator.

This is why the human annotation pipeline sits upstream of everything. You build the golden set first, you measure how well humans agree among themselves, and only then do you point a judge at the same examples and ask: does the judge agree with the humans as much as the humans agree with each other? If it does, you can let it run at scale. If it does not, you keep tuning the judge's rubric — or you accept that this dimension needs a human in the loop. The same logic underpins human-in-the-loop "nugget" annotation: breaking an evaluation down into human-verified factual nuggets makes an LLM judge markedly more accountable, because there is a concrete, human-checked thing to agree or disagree with.

Designing the labelling task and rubric

Most annotation projects fail not at the tooling stage but at the guideline stage. Annotators are almost never the problem; ambiguous instructions almost always are. So the single highest-leverage thing you can do is write a crisp rubric before anyone labels a single example.

A good rubric has three properties. First, it defines each label in one or two plain sentences, in the language your annotators actually work in — this matters when your India team writes to a Chennai reviewer and your UK team writes to a Leeds reviewer, and both must land on the same standard. Second, it carries positive and negative examples for every label: here is a clearly relevant answer, here is a clearly irrelevant one, and — crucially — here are two borderline cases with the correct call and a one-line reason. Third, it names an adjudication path for disagreements, so that when two annotators split, there is a defined way to resolve it rather than an argument in a chat thread.

Keep the label space small. A binary "acceptable / not acceptable" with a required free-text reason will almost always give you higher agreement and more useful data than a seven-point Likert scale that annotators interpret differently. If you genuinely need gradations, an ordinal three-point scale (for example: fails, partially meets, fully meets) is about as fine as most teams can hold consistent. Every extra category you add is another place for two reasonable people to diverge.

Here is a minimal labelling schema you can adapt. Expressed as a small JSON contract, it forces you to be explicit about the label space and the required justification before any UI is built:

{
  "task": "answer_relevance",
  "instructions": "Does the answer directly address the user's question, using only supported facts?",
  "labels": [
    { "value": 0, "name": "fails",     "definition": "Off-topic, refuses, or contradicts the source." },
    { "value": 1, "name": "partial",   "definition": "Addresses the question but omits or hedges a key point." },
    { "value": 2, "name": "full",      "definition": "Directly and completely answers, fully supported." }
  ],
  "require_reason": true,
  "gold_check": true,          // this item is a hidden known-answer question
  "guideline_version": "v3"    // version every guideline; agreement drifts when it changes silently
}

Notice the guideline_version field. Guidelines drift. You will clarify an edge case in week three, and every label collected before and after that clarification now lives under a slightly different standard. If you do not version the guideline and stamp each label with the version it was created under, you will one day see agreement mysteriously fall and have no way to explain it.

Measuring agreement with code

Once two or more people have labelled an overlapping set of examples, you can measure whether they actually agree — and you must, because raw percentage agreement is misleading. If ninety per cent of your examples are "acceptable", two annotators who both rubber-stamp everything will show ninety per cent raw agreement while contributing no information at all. Chance-corrected metrics fix this by asking how much better than random the agreement is.

For two annotators labelling categories, use Cohen's kappa. For more than two annotators, or for ordinal and interval labels, use Fleiss' kappa or, more generally, Krippendorff's alpha — the latter handles any number of raters, any measurement level, and missing labels, which is exactly the messy shape real annotation data arrives in. The rough reading is the same across all of them: above 0.8 is near-perfect, 0.6 to 0.8 is substantial and usually workable, 0.4 to 0.6 is moderate and worth investigating, and below 0.4 is poor.

from sklearn.metrics import cohen_kappa_score
import krippendorff

# Two annotators labelling relevance on the 3-point scale above (0, 1, 2)
rater_a = [2, 1, 0, 2, 2, 1, 0, 1, 2, 0]
rater_b = [2, 1, 1, 2, 2, 0, 0, 1, 2, 0]

# Cohen's kappa — two raters, chance-corrected
kappa = cohen_kappa_score(rater_a, rater_b)
print(f"Cohen's kappa: {kappa:.3f}")

# Krippendorff's alpha — any number of raters, handles gaps (np.nan) and ordinal data
# reliability_data: one row per rater, one column per item
reliability_data = [
    [2, 1, 0, 2, 2, 1, 0, 1, 2, 0],
    [2, 1, 1, 2, 2, 0, 0, 1, 2, 0],
]
alpha = krippendorff.alpha(
    reliability_data=reliability_data,
    level_of_measurement="ordinal",
)
print(f"Krippendorff's alpha: {alpha:.3f}")

# Rough reading:  >0.8 near-perfect  ·  0.6-0.8 substantial  ·  <0.4 poor (fix the guideline)

The most important habit is what you do with a low number. The instinct is to blame the annotators. Resist it. A low kappa is far more often a symptom of an ambiguous guideline — a label whose definition two careful people read differently. When agreement is poor, pull the specific items where annotators split, read the reasons they wrote, and you will almost always find a fixable gap in the rubric. Rewrite the guideline, bump its version, and re-measure. Agreement is a diagnostic for your instructions, not a report card on your people.

Choosing a tool

You do not need a bespoke labelling UI to start; you need something that captures labels, reasons and annotator identity cleanly, and — for teams in India and the UK — something you can self-host so data never leaves your infrastructure. Three self-hostable options cover most builder needs — two of them open-source, one commercial. The right choice depends less on features on paper and more on who is doing the labelling and how they think.

Tool	Best fit	Strengths	Trade-offs	Self-host
Argilla	LLM / NLP feedback loops close to a Python eval stack	Purpose-built for text and LLM workflows; clean Python SDK; pairs naturally with judge calibration and dataset curation	Narrower than a general labelling platform; newer ecosystem	Yes
Label Studio	General-purpose starting point for mixed data types	Very flexible across text, audio, image; large community; configurable interfaces; easy to hand to non-technical annotators	Flexibility means more configuration up front; can feel heavy for a tiny task	Yes
Prodigy	A single expert or small in-house team, scriptable workflows	Fast keyboard-driven labelling; active-learning patterns; scriptable recipes; low overhead for a focused labeller	Commercial licence; developer-centric rather than for large casual crowds	Yes

A common and sensible progression: prototype with Label Studio because it will accept whatever data shape you throw at it, then move the LLM-specific feedback loop into Argilla once your eval stack matures and you want tighter Python integration. Prodigy earns its place when one experienced person is doing careful, high-throughput labelling and wants to script the flow rather than click through a web form. For data exploration and slicing your unlabelled pool before you decide what to annotate, a tool like Lilac is a useful companion to any of the three — it helps you see clusters and duplicates so you do not waste annotation budget on near-identical examples.

Whatever you choose, self-hostability is the non-negotiable line for regulated data. All three can run on your own servers, which is what makes them viable for teams handling personal data under DPDP or GDPR. We will come back to that.

Sampling and the production flywheel

The fastest way to waste an annotation budget is to sample randomly from production traffic. Most production traffic is easy and repetitive; a random sample buries your annotators in near-identical happy-path cases while the interesting failures — the ones that actually move your quality metrics — appear a handful at a time, if at all.

Sample deliberately instead. Stratify by segment so that every user type, language, and product surface is represented in proportion to how much you care about it, not how much traffic it happens to generate. Stratify by failure mode so that each known way your system goes wrong has enough examples to measure. Then oversample the edge cases: the low-confidence traces, the ones where a lightweight heuristic or the model's own uncertainty flags a possible problem, the tail of unusual inputs. A golden set that is deliberately skewed towards hard and diverse cases teaches you far more than one that mirrors raw traffic.

This is where the flywheel turns. Your production logs are not just telemetry; they are a continuous stream of labelling candidates. The traces where your automated judge is uncertain, where users retried or gave negative feedback, where a downstream tool call failed — these are precisely the examples worth putting in front of a human. Label them, fold the verdicts back into the golden set, and your ground truth grows in exactly the directions your system is weakest. Our piece on error analysis from production logs goes deeper on how to mine that stream systematically, and once your golden set is stable you will want to wire it into evals in CI so regressions get caught before they ship.

Adjudication, gold questions and quality control

Two annotators will disagree, and that is healthy — the disagreements are where you learn. What matters is having a defined adjudication step: when raters split on an item, a third, more experienced reviewer makes the final call and, ideally, records a one-line reason that then feeds back into the rubric as a new borderline example. Adjudication is not overhead; it is how your guideline gets sharper over time.

To keep quality honest at scale, seed your annotation queues with hidden gold questions — items whose correct answer you already know, mixed invisibly into the stream. If an annotator starts missing gold questions, it is an early signal of fatigue, a misunderstanding of the guideline, or simple inattention, and you can intervene before their labels pollute the golden set. Gold questions are also the fairest quality metric you have, because they measure against a known answer rather than against other annotators who may share the same blind spot.

Around those two mechanisms sit the ordinary operational realities that decide whether a pipeline survives contact with a real team. Annotator training is a real cost and a real investment — plan a calibration session where everyone labels the same twenty examples and you discuss the splits before production labelling begins. Throughput versus cost is a genuine tension: more annotators per item gives you higher confidence and lets you measure agreement, but it multiplies cost, so most teams double-label a subset for agreement and single-label the rest. And guideline drift is the slow killer — the standard shifts under you as the team clarifies edge cases, which is exactly why the versioning discipline from earlier is not bureaucratic box-ticking but the thing that keeps your March and September numbers comparable.

Data residency for India and the UK

If your annotation data contains personal information — and production traces very often do — then where that data lives and who touches it is a compliance question, not just an engineering one. India's Digital Personal Data Protection Act and the UK GDPR both impose obligations around lawful basis, purpose limitation, and retention, and both make you responsible for what your processors (including annotation vendors) do with the data.

The practical implications for a builder team are concrete. Prefer self-hostable tools — the reason Argilla and Label Studio (both open-source) and the commercial, self-hostable Prodigy all run on your own infrastructure matters here, because raw production traces never have to leave your environment or cross a border to a third-party SaaS. Pseudonymise or redact personal data before it reaches annotators; most relevance and quality labels do not need a real name or account number to be made correctly. Keep Indian data and UK data in the appropriate regions rather than pooling everything in one convenient bucket. And record the boring-but-essential paperwork: a lawful basis for processing, a retention window after which labels and source data are deleted, and an audit trail of who labelled what, under which guideline version, and when.

None of this needs to be heavy for a small team. A self-hosted Label Studio instance in the right region, a redaction pass in your sampling script, and a versioned guideline with per-label provenance will satisfy the substance of both regimes for most builder-scale workloads. The point is to design residency in from the start, because retrofitting it after you have shipped personal data to a foreign SaaS annotation tool is expensive and, occasionally, not fully reversible.

Pitfalls to avoid

A handful of mistakes account for most failed annotation pipelines, and all of them are avoidable once named.

Do these five things in reverse — version your guidelines, sample deliberately, calibrate every judge, read low agreement as a rubric signal, and keep regulated data on your own infrastructure — and you have a pipeline that produces ground truth you can actually stake decisions on. That is the whole game. A golden set is not a dataset you build once; it is a living asset you tend, and the teams that tend it well are the ones whose eval numbers mean something twelve months later.