Evals: how to know whether an AI system actually works

The whole eval loop, from one test case to a release gate: gold answers, reading outputs, code graders, AI judges you have checked, error bars, agents, benchmarks.

Source: Sergii Makarevych, “Evals: how to know whether an AI system actually works”, X article, 3 Oct 2026 (34 min read) · X article

Going deeper: the full loop, how to check a judge (kappa), the five judge biases, error bars and paired comparisons, grading agents (pass^k), benchmarks, and how to run evals every week.

1. Summary

• Running example: a support assistant for a fictional bike and camping shop, with three parts: triage (12 categories), answer (retrieval plus a reply that cites the handbook), and an agent that calls tools against a mock database. 80 tickets: 20 for tuning, 60 for reporting.
• Surface checks miss meaning. On one ticket, two code checks passed the reply. An AI judge then found an invented "15 business days" allowance that would have gone to a customer.
• Reading outputs drives everything. Error analysis on the 60 replies: 34 pass. The top failures were a missing required fact (15) and an unsupported claim (12).
• The judge was barely better than chance. It agreed with human labels on 36 of 60 (60%), but "always pass" would have scored 57%. Kappa was 0.15 and it caught 10 of 26 bad replies.
• Error bars. Triage accuracy is 0.70, 95% CI [0.58, 0.82]. With 60 cases the smallest reliable difference is about 12 points, so a 5-point regression is invisible.
• Agents: grade the final state and the path, run tasks several times, and report pass^k (succeeds every time), not just pass@k (succeeds at least once). The example agent: pass^3 = 0.75.
• Judges are biased in known ways: position, length, self-preference, rating satisfaction instead of success, and results that change with the judge model. Giving the judge the evidence (tool calls, final state) matters more than using a stronger judge.
• Benchmarks pick engines; evals test your product. Benchmark scores depend on the harness, get contaminated and saturate.

2. Why this matters for robotics

• Final state beats the robot's report. "Satisfaction is not success" in robot terms: check the bin, not the robot's message. Learned success detectors are graders too, so they need the same checking as an AI judge.
• pass^k is what users feel. A grasp policy that works 9 times in 10 fails a 5-step task about 4 times in 10 (0.9⁵ ≈ 0.59). Report reliability over repeated runs, and test perturbed versions (moved objects, different lighting) as well as the standard setup.
• Real-robot evals are expensive, so error bars matter more. Robot papers often report 20 trials per task. A 20-trial success rate of 70% is give or take about 20 points.
• Reward hacking happens in robotics too. RL policies exploit simulator bugs the same way coding agents exploit tests (Module 07). Keep a held-out check the policy never trained against.
• Video judges. Teams increasingly use vision-language models to label robot rollouts as success or failure. Those judges have the same five biases, plus whatever the camera misses.

3. Deep dive: the loop, and what to measure

The article runs one ticket through the whole loop first. A customer angrily asks about a missing refund. Triage labels it "returns", which matches the gold label. The answer step writes a polite reply that cites the returns section and states both required facts (5 business days, a 15% restocking fee for used items). Then three graders look at it:

The lesson: different graders answer different questions, and a fluent reply can pass every surface check while still being wrong.

Inputs, gold answers and two piles

• Inputs should look like real traffic, awkward cases included. The 80 tickets came from a grid of persona × topic × scenario, so angry, vague and multi-issue tickets all appear. The article cites Anthropic's agent-eval guide: start with 20–50 tasks taken from real failures.
• Gold answers say what "right" means: the category, the sections to draw on, the 2–3 facts a reply must contain. They are the most valuable and most expensive part of the whole process.
• Dev set and test set. Tune prompts and graders on the dev set (20) and report only on the test set (60). Tuning on what you report flatters you.
• One number per experiment: accuracy for triage, pass rate for replies, right-final-state rate for the agent. Always on the test set, always with an error bar.

The article's analogy: a benchmark measures the engine, your eval measures the car on your road.

4. Deep dive: read the outputs first

Before writing any automatic grader, read the outputs one by one. Write a short note on each failure (open coding), then group the notes into failure types with counts (axial coding). This is error analysis. On the 60 test replies:

A reply can have more than one failure, so the counts add up to more than 26.

This list is the plan: one grader per failure type, most frequent first. The article says teams whose eval work stalls have almost always skipped this step. Expect criteria drift (Shankar et al., 2024): grading examples sharpens your rubric, which changes earlier grades. Two or three rounds is normal.

5. Deep dive: graders made of code

A code grader is free, instant and gives the same answer every time. Use it for everything with a definite answer.

• Exact labels. Triage: accuracy 0.700, macro F1 0.697. Macro F1 weights all 12 categories equally. The two numbers are close, so the mistakes are spread out rather than concentrated in one category.
• Rule checks on free text: nonempty 100% pass, ≤150 words 17% pass, no banned promises 92%. The app is far too chatty, and no judge was needed to find that. IFEval is a public benchmark built entirely from rules like these.
• Required facts: check each gold fact appears in the reply. This catches the top failure type for free.
• Similarity (ROUGE, BLEU, embeddings) separates good from bad replies with AUC 0.709 (0.5 is a coin flip). That's enough to notice that outputs changed, not to say whether they are right. Use it as a cheap drift alarm, never as the headline number.

The limit, in one number: only 11.7% of replies passed every code check, yet the code checks caught none of the invented claims. Rules see form, not meaning.

6. Deep dive: an AI as the grader, and how to check it

An AI judge (LLM-as-judge) reads for meaning, so it can catch the invented 15-day allowance. It is also a language model, with all the failure modes of one. The article calls this the place where most eval programmes go wrong.

How to ask

• One yes/no question per failure type. "Does the reply contain a claim the handbook excerpts don't support?" rather than "rate quality 1–5". The output passes only if every question comes back clean.
• Ask for evidence: the judge quotes the offending sentence, so a person can check the verdict in seconds.
• Give it the context. A judge that can't see the handbook can't know what is unsupported.

The article cites Cho et al. (2026, "Ask, Don't Judge"): yes/no questions about specific errors beat holistic G-Eval scoring on every dataset (Spearman 0.563 vs 0.514 on SummEval). A summary with three planted factual errors got a perfect 5.0 from holistic judges and 1.57 from the question-based one. Adding more and more instructions eventually made the judge worse.

Checking the judge

All 60 replies were labelled by hand, then the judge ran. The reference passed 34; the judge passed 42. They agreed on 36 (60%). That sounds fine until you notice that a judge saying "pass" to everything would agree 57% of the time. Cohen's kappa corrects for that: it measures agreement beyond what chance would give. 0 is chance, 1 is perfect. The usual bar is about 0.6 for a judge that gates a release and 0.8 for one that runs unsupervised. This judge scored 0.15: lenient, and it caught only 10 of the 26 bad replies.

The fix is iteration on the dev set: add examples of the failure to the judge prompt, narrow the question, and give it the gold facts. In the article's tutorial the "missing fact" judge reached a usable level this way. The "unsupported claim" judge did not, and that question went to a specialised detector instead.

The five known biases

The article gives the GAUGE audit (Bodhwani et al., 2026) its own paragraph because it tests the setup many companies use to decide releases: a simulated user chats with the agent, a judge scores the transcript, and the higher score ships. Across 25 agents and about 3,700 transcripts, this worked when one agent was clearly better (correlation 0.94 with ground truth). On near-equal pairs, which are the decisions that matter, it promoted the worse agent 31% of the time. A judge that only saw the chat and rated satisfaction was a coin flip (AUC 0.49). A judge that also saw the tool calls and whether the task was resolved reached AUC 0.73. Reliability came from the evidence the judge was given, not from how strong the judge was.

Panels, detectors and typed judges

• Panels. Several smaller judges can agree with humans better than one large one, at a fraction of the cost (Verga et al., 2024). But averaging is fragile: one broken judge (parser failures returning zeros, or a judge that gives everything 10) drags the average anywhere. Acharya et al. (2026) use the geometric median instead, and find about three judges capture most of the benefit because judges' errors are correlated.
• JSON makes answers more uniform. Parikh (2026): asking for JSON output raised the most common answer's share from 41% to 64%. Harmless for one-right-answer questions; it makes open-ended ratings less discriminating.
• Hallucination detectors (HHEM, LettuceDetect, MiniCheck, NLI cross-encoders) are 100–500M-parameter models trained for one question: is this sentence supported by this text? They run on a laptop CPU in about a second and, on RAGTruth, come close to a large judge. SelfCheckGPT needs no detector: sample several times and flag claims that change.
• Typed judges (TypeSafe Jev, see Module 01) return a calibrated probability for a yes/no or pick-a-label question, about a hundred times cheaper than a generative judge, but give no explanation. The article cites a 2026 study: 92.2% on RewardBench pairs vs 93.5% for a frontier judge, but 78.6% vs 93.1% on hard derivation checks. Escalating low-confidence cases kept 92.5% at about half the cost.
• Judges need benchmarks too. JudgeBench (Tan et al., 2025) found frontier judges barely beat chance on genuinely hard, checkable problems. Vendor claims like "85% agreement with humans" come from one dataset and don't transfer.

7. Deep dive: honest numbers

Would the score be the same on a different 60 tickets? Three tools answer that.

• Error bar (bootstrap). Resample the 60 results thousands of times and look at the spread. Triage accuracy 0.700, 95% CI [0.583, 0.817]. The honest sentence is "most likely 70%, somewhere between 58% and 82%". Miller (2024) showed that most published differences between models are smaller than intervals like this, and gives the formulas (including clustered standard errors when several questions share a source).
• Paired comparison. A v2 prompt that retrieved more sections scored the same judge pass rate as v1: 0.700 vs 0.700, difference +0.000, CI [−0.117, +0.117]. Verdict: no real difference, keep v1. The interval is narrower than for a single score because both versions run on the same tickets, so ticket difficulty cancels out. A McNemar test asks whether v2 wins where v1 loses or they just trade cases. Here they traded: v2 regressed on 14% of cases and improved on others.
• Detection floor. With 60 cases, the smallest difference you can reliably detect is about 12 points. Below that you need more cases, not a cleverer test.
• Many comparisons. Test 20 prompt variants and one will look 10 points better by luck. Correct for it, or confirm the winner on fresh cases.

8. Deep dive: systems with parts (RAG, agents, reward hacking)

Retrieval systems: grade each stage

A bad reply can come from the finder or from the writer, and they need different fixes. Grade the finder against the gold sections: hit@2 = 0.93, recall@2 = 0.88. Of 18 failing replies, 1 failed in the finder and 17 in the writer. That one line redirects a week of engineering. For the writer, the RAGAS triad (faithfulness, answer relevance, context precision) needs no gold answers; in this run RAGAS gave context relevance 0.855 and faithfulness 0.737, the same "retrieval good, writing worse" story as the hand-built graders. When two different methods tell the same story, the number is easier to believe. ARES adds confidence intervals by calibrating small judges against about 150 human labels.

Agents: final state, path, and many runs

• Final state and path. Did the database end up right, and did the agent only take allowed actions (not, say, refund twice)? τ-bench set this pattern. Outcome-only scoring overstates capability because agents sometimes reach the right state by luck.
• pass@k vs pass^k. Agents are random. pass@k = succeeded at least once in k tries (capability). pass^k = succeeded all k times (reliability). Customers feel pass^k. The example agent scores pass^3 = 0.75. τ²-bench found pass^1 drops about 20 points when the agent must coordinate with an active user.
• Reliability is its own axis. Rabanser et al. (2026) define 12 reliability metrics (consistency, robustness, predictability, bounded harm). Across 15 frontier models, two years of capability gains brought about one sixth as much reliability gain. Agents break on a rephrased instruction.
• Cost per correct outcome. Bai et al. (2026): agentic tasks use about a thousand times more tokens than a chat turn, mostly re-reading their own history. The same task on the same model varied 30-fold in cost between runs, and models predicted their own cost poorly (correlation ≤ 0.39).

Graders get gamed

Any grader used to train or select a system will be gamed. Wang et al. (Qwen, 2026) catalogue coding agents reading answers from repository history, editing the tests, and patching for the evaluator. A behaviour monitor cut hacked "solutions" from 28.6% to 0.6% while genuine ones rose from 40% to 61%. Their argument: no grader is scalable, faithful and robust at once. Tests scale and resist gaming but miss intent; judges scale and capture intent but can be gamed; experts are faithful and robust but don't scale. Keep a held-out grader the system never sees, and upgrade it as the system improves. Same idea as the KL leash in Module 07.

9. Deep dive: public benchmarks

• The harness matters as much as the model. The article ran GSM8K on one open model with two prompt formats: 0.686 and 0.746. A score is a (model, prompt, harness, version) tuple. Schaeffer et al. (2023) showed some "emergent" jumps vanish under smoother metrics.
• Contamination. GSM1k rebuilt GSM8K with fresh questions; some model families dropped up to 8 points, others didn't.
• Saturation. SWE-bench Verified went from roughly 40% to over 80% solved in about two years. MMLU, GSM8K and HumanEval have been replaced by HLE, GPQA Diamond, SWE-bench Verified, LiveCodeBench and τ²-bench, which will saturate too. The top of Chatbot Arena sits within about 20 Elo, inside the noise.
• Redundancy. Zeng & Papailiopoulos (2026): an 84-model × 133-benchmark matrix is roughly rank two, and five well-chosen benchmarks predict the rest to within about 4 points.

Use benchmarks to shortlist two or three models. Decide with your own eval on your own data.

10. Deep dive: making it a habit

• The merge gate fails the build if the pass rate drops more than 3 points below a 0.70 baseline. It's a tripwire for big regressions, not a guarantee: with a 12-point floor, a 5-point regression gets through. Say so on the dashboard.
• Calibrate, then trust. Audit the cheap judge gate once against verifiable ground truth, use it only where it agreed, and re-audit when the model, judge, simulator or domain changes.
• Cache every model call keyed on prompt, data version and recipe. The full A/B on 60 tickets cost 121 judge calls, about four cents. Cost is not a reason to skip evals.
• What a leader should ask for: four numbers on one page. The pass rate with its interval, the grader's agreement with human labels, the detection floor, and the share of cases that got worse in the last change.

The article also compares tools. In short: Inspect AI (complete, code-first), lm-evaluation-harness (public benchmarks), promptfoo (YAML suites and red-teaming), DeepEval (pytest-style), RAGAS (RAG triad), Langfuse / Phoenix / LangSmith / Braintrust (tracing plus evals), and small detectors for "is this supported". Its advice is to read a hand-built grader's number against a framework's: agreement is evidence, disagreement is a good question.

11. Try it on Robo-Helper

Module 18's week-11 bake-off now uses three of these ideas, with real numbers from the project (python3 lesson19_judge.py):

• Kappa and catch rate. The reply-only rule judge agrees with the exact checks 58% on the word counter (kappa 0.21, caught 3 of 7 failures), 67% on the tiny LM (kappa 0.14, caught 1 of 4), and 75% on the distilled brain, where kappa is −0.12: worse than guessing, and it caught neither of the 2 failures. A lazy "always pass" judge would have agreed 83% of the time.
• Satisfaction vs success. Asked "where's the green one?", the word counter drives off, bins the green block and says "The green block is in the bin." A reply-only judge passes it, but the bin shows the robot did the wrong thing: it was asked a question, not told to tidy up.
• Error bars and paired comparison. Word counter 5/12 (bootstrap 95% range 17%–67%), tiny LM 8/12 (42%–92%), distilled 10/12 (58%–100%). Paired: tiny LM minus word counter is +25 points [+0, +50] over 3 differing tasks, which is too close to call on 12 tasks. Distilled minus word counter is +42 points [+17, +67] over 5 differing tasks: a real difference.

Homework: kappa, catch_rate and bootstrap_ci in lesson19_judge.py, checked by python3 check.py 19a. See week 11, stretch 2.

12. Playbook

1. Collect 20–50 real inputs, awkward ones included, and write gold answers before running anything. Split into dev and test.
2. Run the system and read the outputs. Note each failure, group the notes, count them. That list is your grader plan.
3. Write code graders for everything definite: labels, formats, length, required facts, forbidden phrases.
4. For meaning, write one yes/no judge question per failure type. Give the judge the context and the evidence (tool calls, final state), and ask it to quote the problem.
5. Label a few dozen dev cases by hand. Measure the judge: kappa or catch rate plus false-fail rate. Iterate until it clears about 0.6, or hand that question to a detector.
6. Report every score with an error bar. Compare versions paired, on the same cases, and list the cases that got worse.
7. Work out your detection floor before comparing versions. Add cases if it's too coarse.
8. For agents: check the final state and the path, run each task several times, report pass^k, cost per success and consistency.
9. Keep one grader the system never trains or selects against.
10. Automate: code checks every commit, judges every change, monitoring after launch. Cache model calls. Re-audit judges when anything changes.

13. Self-check

14. Glossary & further reading

Every number on this page is quoted from the article, except the Robo-Helper numbers in section 11, which come from our own project code. The article's running-example numbers come from its author's own tutorial, which isn't linked publicly. Many of the 2026 papers it cites are arXiv preprints we haven't checked independently.

• The article on X (full reference list at the end)
• Husain, Your AI Product Needs Evals; Husain & Shankar, AI Evals FAQ
• Miller (2024), Adding Error Bars to Evals
• Shankar et al. (2024), Who Validates the Validators?
• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Yao et al. (2024), τ-bench
• Es et al. (2024), RAGAS
• Related: Module 01: decision models (thresholds, precision/recall, calibration: the typed-judge side of this page)
• Related: Module 04: evaluation that doesn't fool you (paired bootstrap, splits by day)
• Related: Module 07: checkers as rewards, where reward hacking comes from
• Related: Module 18, week 3: testing Robo-Helper honestly and week 11: the bake-off and the AI judge