Error Analysis First: Build LLM Evals From Production Logs

What you need to know

The default failure of LLM evaluation in 2026 is not bad metrics — it is premature infrastructure. A team ships a RAG feature or a support agent, stands up a dashboard of generic scores (hallucination rate, toxicity, relevance), wires an eval framework into CI, and discovers weeks later that none of it maps to how the product actually fails in front of users. The fix is a method popularised by Hamel Husain and Shreya Shankar — through Husain's writing on evals and the pair's widely cited AI evals FAQ (updated January 2026) — called error analysis, and it inverts the usual order of work. You read your production traces first, write down what went wrong in plain language, count the failures, and only then automate evals for the failure modes that actually occur. Their FAQ is blunt on both the priority and the price: error analysis is, in their words, "the most important activity in evals", and the entry cost is 30 minutes spent manually reviewing 20–50 outputs whenever you make a significant change. That half-hour, done honestly, routinely beats weeks spent building a speculative eval harness for failures your product does not have.

• Log complete traces — the full record from user query to final response, including every tool call and retrieval in between.
• Read 20–50 of them yourself, in about 30 minutes, writing a free-form note on each — this is open coding.
• Cluster the notes into a failure taxonomy with counts per failure mode — this is axial coding, and the counts set your priorities.
• Automate only the top failure modes, choosing between code assertions, golden-set tests and LLM judges per mode.
• Prefer binary pass/fail judgements over 1–5 scores — they are faster, more consistent and easier to calibrate.
• Repeat on a cadence — a weekly half-hour pass, plus a fresh sample after every significant change.

What a trace is and what to log

Error analysis needs raw material, and the raw material is traces. The word comes from ordinary software engineering: as Husain puts it in Your AI Product Needs Evals, a trace is "a log of a sequence of events", such as a user session or a request flowing through a distributed system. For an LLM product, a trace is the complete record of one interaction: the user's query, the context your system assembled, every retrieval and its results, every tool call with arguments and outcomes, and the final response — plus, ideally, what happened next. A bare prompt/response pair is not a trace; it is the last page of a story with the plot missing. When a support bot gives a wrong answer, the cause is usually upstream — a retrieval that pulled the wrong document, a tool call that timed out and was silently ignored — and you can only see that if you logged the whole sequence.

For agents the bar is higher. An agent's trace should capture the entire workflow, not just the model turns: which tools were called in which order, what each returned, any human-approval steps and what the human decided, and any writes the agent made to a database or external system. If your agent files a ticket, updates a CRM record or sends an email, the trace should record that action and its result. Teams in Bengaluru and London alike tend to discover the same thing in their first review session: the failures that matter most are almost never in the model's prose — they are in the plumbing around it.

A practical logging checklist:

• A unique trace_id and a session_id linking multi-turn conversations.
• The raw user query, verbatim, plus any system-assembled context.
• Model name and prompt version — you cannot debug what you cannot attribute.
• Every retrieval: query used, source documents, chunk count.
• Every tool call: name, arguments, result status, and the result payload (or a reference to it).
• Human-in-the-loop events: approvals, rejections, edits.
• Database or external writes the agent performed.
• Outcome signals: did the user follow up, rephrase, escalate, or abandon?

A minimal JSON shape that captures this — adapt the field names to your stack:

{
  "trace_id": "tr_20260714_183302_9f4c",
  "session_id": "sess_8821",
  "timestamp": "2026-07-14T18:33:02Z",
  "user_query": "Where is my refund for order #58421?",
  "context": {
    "account_region": "UK",
    "model": "gpt-5.2-mini",
    "prompt_version": "support-v14"
  },
  "steps": [
    { "type": "retrieval", "query": "refund status order",
      "source": "refund_policy_v3.md", "chunks": 4 },
    { "type": "tool_call", "name": "get_order",
      "args": { "order_id": "58421" }, "result_status": "ok" },
    { "type": "tool_call", "name": "get_refund_status",
      "args": { "order_id": "58421" }, "result_status": "timeout" },
    { "type": "llm_response", "tokens_out": 212 }
  ],
  "final_response": "Your refund was processed on 12 July and should...",
  "human_actions": [],
  "db_writes": [],
  "outcome": { "user_followed_up": true, "escalated_to_human": true }
}

Notice what this single illustrative trace already tells a reviewer: the refund-status tool call timed out, yet the final response confidently states a processing date. No generic hallucination score would have surfaced that; thirty seconds of reading did.

The manual review pass

With traces flowing, the first real step is a manual review pass — and the operative word is manual. Sample 20–50 traces. A useful split is half random (so you see the true distribution, including the boring successes) and half failure-biased: traces where the user rephrased, escalated to a human, gave a thumbs-down, or abandoned the session. Pull them into whatever lets you read fastest — a spreadsheet with one row per trace works fine on day one, and a purpose-built annotation view is worth building soon after. Husain and Shankar's FAQ calls a custom annotation tool "the single most impactful investment you can make" for an eval workflow, and Husain's earlier essay gives the underlying principle: "You must remove all friction from the process of looking at data."

For each trace, write a short free-form note about what went wrong — or note that nothing did. This is open coding, a term borrowed from qualitative research: you are deliberately not forcing observations into predefined categories, because predefined categories are exactly the assumption error analysis exists to test. Two disciplines make the notes far more useful. First, record the first failure you observe in the trace, because upstream errors cascade — a bad retrieval makes every downstream judgement suspect, so blaming the final response wording is noise. Second, be concrete: "cited the v2 policy document, current is v3" ages far better than "answer seemed off".

Here is a worked example — a fictional but realistic trace from an e-commerce support bot, with the reviewer's note attached. It is illustrative, not from a real deployment:

TRACE tr_0043 -- 14 Jul, 09:12 IST
User: "I was charged twice for my June invoice. Can you refund one?"
Retrieval: billing_faq.md (3 chunks; none mention duplicate charges)
Tool call: get_invoices(customer_id="c_2291") -> June + July invoices, both "paid"
Final response: "I can see your June invoice was paid successfully.
Our refund policy allows refunds within 14 days of purchase..."

REVIEWER NOTE (open coding):
Bot answered a different question. User reported a DUPLICATE charge;
bot answered about refund eligibility. Retrieval query was built from
the word "refund", so it pulled the refund policy instead of billing
troubleshooting. No payment-events lookup was ever called. First
failure: query construction, not the final response.

Thirty minutes of this is uncomfortable in a way dashboards are not — you will watch your product misunderstand real people in specific, avoidable ways. That discomfort is the point. It is also why the reading cannot be delegated to a model: as the FAQ's guidance on automation puts it, you should always read the raw traces yourself at the start, because that is how you discover new failure types and build intuition about your data — an LLM does not have your product context or your tribal knowledge, and it will happily summarise away the surprise that would have changed your roadmap.

From notes to a failure taxonomy

A pile of free-form notes becomes useful when you cluster it. This second pass is axial coding: read through your notes, group the ones describing the same underlying problem, give each group a short name, and count how many traces fall into each. The output is a failure taxonomy — the single most decision-relevant artefact in this whole process, and the step Husain and Shankar's error-analysis FAQ entry flags as the most important. Here is what one might look like for our illustrative support bot after a 50-trace pass:

Failure mode	Count (of 50)	%	Example
Wrong question answered	11	22%	User reports duplicate charge; bot explains refund policy
Unsupported claim	9	18%	States a refund date when the status tool call timed out
Missing escalation	7	14%	Frustrated user asks for a human; bot keeps troubleshooting
Stale policy citation	5	10%	Quotes the superseded v2 returns policy
Buried answer	4	8%	Correct refund date hidden in a 300-word reply
No failure observed	14	28%	—

The counts are the whole argument. Before this table exists, every eval idea sounds equally sensible and the loudest stakeholder wins. After it exists, the conversation changes shape: query construction and unsupported claims account for 40% of all traces reviewed, so they get fixed and evaluated first; response length, which someone would certainly have built a metric for, turns out to be an 8% problem. The taxonomy tells you what to fix, in what order, and — crucially for the next section — what your first automated evals should test. It also gives you a baseline: after you ship a fix, re-run the same sampling and see whether the count actually dropped.

Once you have manually coded 30–50 traces, this clustering step is the one place an LLM can legitimately help early: hand it your raw notes and ask for proposed groupings, then review and rename the clusters yourself. The FAQ draws exactly this line — first-pass axial coding can be accelerated with a model once your own open coding has happened, but validating the taxonomy stays a human job, because models tend to merge failure modes that look linguistically similar and are causally distinct.

From taxonomy to automated evals

Only now — with named, counted failure modes in hand — does it make sense to automate. For each of your top failure modes, pick the cheapest evaluator that can detect it reliably. There are three tiers, and the right choice is a property of the failure mode, not a house style:

Failure mode (from taxonomy)	Evaluator type	Why it fits
Stale policy citation	Code assertion	Deterministic: check the cited document version against the current one
Missing escalation	Code assertion	Rule on the trace itself: escalation-intent detected but no handoff tool call made
Wrong question answered	Golden-set test	Curate real queries with known-correct intents; assert the routing/retrieval choice
Unsupported claim	Binary LLM judge	Needs semantic comparison of the reply against tool output; no regex can do it
Buried answer	Code assertion	Cheap structural checks: length cap, answer-first ordering

Code assertions come first because they are free to run and impossible to argue with. They are ordinary tests — Husain's three-level framing puts them at Level 1, run frequently and cheaply — except that their content comes from your taxonomy rather than your imagination. If you struggle to think of assertions, his advice is to critically examine your traces and failure modes; after a review pass, you will have the opposite problem. A pytest-style example targeting the "unsupported claim" mode's most common trigger:

# test_refund_replies.py
from app.agent import run_support_agent

def test_no_refund_date_when_status_tool_fails():
    """Failure mode: 'Unsupported claim' (18% of July audit).
    The bot must not state a refund date if get_refund_status failed."""
    trace = run_support_agent(
        "Where is my refund for order #58421?",
        tool_fixtures={"get_refund_status": "timeout"},
    )
    reply = trace.final_response.lower()
    assert "processed on" not in reply, \
        "Agent asserted a refund date without tool confirmation"
    assert any(p in reply for p in ("couldn't confirm", "unable to confirm",
                                    "check back")), \
        "Agent should acknowledge the status lookup failed"

Golden-set tests handle failure modes where correctness is known but not mechanically checkable — curated real inputs with expected behaviours, run on every change. We cover how to build and maintain these in our guide to golden sets and judges. LLM judges are the last resort, reserved for failure modes that genuinely require semantic judgement — and they should be binary. Husain and Shankar's argument is that binary evaluations force clearer thinking and more consistent labelling: on a 1–5 Likert scale the gap between a 3 and a 4 means different things to different annotators (and to the same annotator on different days), detecting a real change requires much larger samples, and both humans and judge models hedge towards the middle. A pass/fail question can be checked against human labels directly — which is precisely how you calibrate a judge, a process Shankar and colleagues formalised in Who Validates the Validators? and which we walk through in our LLM-as-a-judge calibration guide. A minimal binary judge prompt for the unsupported-claim mode:

You are reviewing one customer-support conversation.

Failure mode under test: "Unsupported claim" -- the assistant states a
refund date, amount, or status that is not present in the tool results
provided below.

Conversation transcript:
{transcript}

Tool results:
{tool_results}

Question: does the assistant's final reply contain any refund date,
amount, or status NOT supported by the tool results?

Answer with exactly one word on the first line: PASS or FAIL.
On the second line, give a one-sentence reason citing the specific
claim and the tool result (or its absence).

Making it a habit

Error analysis is not a launch ritual; it is a cadence. The version that survives contact with a real team calendar looks like this. Weekly: one 30-minute pass over 10–20 fresh traces, biased towards outliers — the escalations, the thumbs-downs, the sessions that ended abruptly. After every significant change — a new model, a rewritten system prompt, a new tool, a retrieval overhaul — a fuller 20–50 trace sample, because changes shift the failure distribution and last month's taxonomy silently stops describing reality. Periodically, every few weeks for a product under active development, a deeper audit of 100 or more traces until you hit saturation again. Husain and Shankar's FAQ sketches essentially this rhythm, and the striking thing is how small it is: the weekly habit costs one meeting slot, which is roughly what many teams spend arguing about a dashboard nobody trusts.

Each cycle feeds the automation loop. A new failure mode that shows up twice gets a note; one that shows up five times gets a named category, a fix, and an eval; and every eval you write joins the regression suite that runs in CI on each prompt or model change — the wiring for that is covered in our guide to running evals in CI. The taxonomy becomes a living document: failure modes get retired when their counts hit zero for a few consecutive cycles, and the retired eval stays in CI as insurance against regression.

There is also a graduation point. Once your top failure modes are automated, counts are stable, and manual passes mostly confirm what the evals already caught, the marginal value of reading shifts from discovery to monitoring — and that is when investing in automated drift detection over your production scores makes sense, as covered in our production drift-detection guide. Drift monitoring built before error analysis watches metrics that may mean nothing; built after, it watches the specific failure modes you have already proven matter.

Common traps

Three traps account for most failed attempts at this method, and all three are attempts to skip the uncomfortable part.

The third trap is over-sampling happy paths. If you review only a uniform random sample, and your product succeeds 70–80% of the time, most of your precious reading time is spent admiring successes — and the taxonomy under-represents exactly the rare-but-severe failures (a wrong refund amount, a missed escalation from a distressed user) that damage trust most. Keep a random slice so you know the true failure rate, but deliberately bias the rest of the sample towards outcome signals: escalations, follow-ups, abandonments, negative feedback. In regulated settings — a UK firm answering FCA-adjacent queries, an Indian lender's collections bot — it is worth maintaining a standing rule that every escalated trace gets read, regardless of sampling.

Start with the traces, not the tooling

The error-analysis-first method is unusual among engineering practices in that the hard part is not technical. Logging a JSON trace, clustering notes into a table and writing a pytest assertion are all afternoon-sized tasks. The hard part is the discipline of looking: sitting with 50 real interactions, writing down what actually went wrong, and letting the counts — rather than intuition, vendor defaults or the loudest voice in the sprint review — decide what gets built next. Teams that do it tend to report the same sequence: surprise at the first session, a taxonomy that contradicts their assumptions, and evals that finally fail when the product does.

If you run LLM features in production, the next step fits in tomorrow's diary: pull 20 traces, read them, and write one note per trace. You will know more about your product's real failure modes by lunch than most eval dashboards will ever tell you. And if you have already done this — built a failure taxonomy from production logs, calibrated a binary judge against human labels, watched a failure mode's count fall after a fix — that is precisely the concrete, verifiable work a hiring manager wants to see on a Builder profile: not "I did evals", but the taxonomy, the counts, and the regressions it caught.

Sources and further reading: Hamel Husain and Shreya Shankar, the AI evals FAQ (updated January 2026), including the entries on why error analysis matters and what can be automated with LLMs; Hamel Husain, "Your AI Product Needs Evals"; Shankar et al., "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences" (arXiv:2404.12272).