Evaluation-driven development & red teaming

Decide how you'll measure success before you build, check that your test questions haven't leaked into training, and attack your own model on purpose so you find its weak spots before users do.

Source: Suraj Sharma, X post, stage 10

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. What this stage covers

• Learn: contamination detection, dynamic evals, adversarial robustness, LLM-as-a-judge calibration.
• Practice: build an automated attacker that systematically searches for inputs your classifier gets wrong, using a tree of candidate rewrites.
• Why (the post's claim): static benchmarks are easy to game, so you need adversarial harnesses that keep attacking your own models. Mostly right. See the reality check for the nuance.
• Builds on: Module 01 for thresholds, cross-validation and calibration of judges, and Module 04 for honest splits, cutoff dates, paired bootstrap and slices. This page doesn't repeat them.

2. Concepts, explained

Contamination detection

A benchmark is contaminated when its items, or near-copies, are in the training data. Then the score measures memory, not skill. There are three families of check:

• Overlap with the training corpus (needs the data). GPT-3 (Brown et al., 2020) flagged a test item if it shared any 13-word sequence with the training set. It's fast and catches verbatim copies. It misses paraphrases, translations, and copies with the numbers changed. The demo below shows why.
• Black-box statistical tests (no data needed). Oren, Meister, Chatterji, Ladhak & Hashimoto (2023) noticed that a benchmark's items are exchangeable: if the model never saw it, every ordering should be about equally likely. A model that memorised the file will find the canonical order much more likely than shuffled orders. That gives a test with a p-value.
• Fresh mirror sets. Write new items matched in style and difficulty, and compare. Zhang et al. (2024) built GSM1k to mirror GSM8k and saw accuracy drops of up to 8% for some model families, with frontier models showing little sign of overfitting. The gap is the evidence.

overlap(item, corpus, n) = |ngrams_n(item) ∩ ngrams_n(corpus)| / |ngrams_n(item)|
flag if any shared n-gram (n = 13 in GPT-3), after lowercasing and stripping punctuation
gap = acc(public benchmark) − acc(fresh mirror set)     # large gap ⇒ suspect memorisation

Dynamic evals

A static test set slowly leaks: into web crawls, into synthetic data made by other models, and into your own decisions as you tune against it. Dynamic evals keep the test fresh:

• Time-based refresh. LiveBench (White et al., 2024) regularly adds questions from recent sources and scores them against objective ground truth. Only items newer than a model's training cutoff are guaranteed unseen. This is Module 04's cutoff-date split applied to benchmarks.
• Procedural generation. Templates or programs that produce new instances (new numbers, new entities, new code tasks) with known answers. Cheap and unlimited, but the template itself can be learned.
• Live human comparison. Chatbot Arena (Zheng et al., 2023) collects head-to-head votes on new user prompts. Fresh, but noisy and tied to who's voting.
• Adversarial refresh. The red-team loop below is itself a dynamic eval: it generates new hard cases for the current model.

The cost: scores from different months aren't directly comparable, because the questions changed. Keep a frozen regression set alongside the fresh one.

Adversarial robustness

Average-case accuracy says little about worst-case behaviour. Adversarial evaluation asks: if someone searches for a failure, how quickly do they find one?

• LMs red-teaming LMs. Perez et al. (2022) used one language model to generate test cases for another and a classifier to score the replies, finding tens of thousands of offensive replies in a 280B chatbot. They compared zero-shot generation, few-shot and RL-trained attackers, trading off diversity against difficulty.
• Search-based black-box attacks. PAIR (Chao et al., 2023) has an attacker LLM refine candidates from the target's responses. TAP (Mehrotra et al., 2023) turns that into a tree: branch several refinements, prune ones that drift off the test goal before querying the target, keep the most promising. This is what the post means by "tree-of-thought prompting". TAP reported success on more than 80% of test goals against the GPT-4-class models it tried, and found it could get past a guardrail classifier (LlamaGuard).
• White-box attacks. Gradient-guided searches over input tokens (Zou et al., 2023, GCG) need weights and are useful for stress-testing open models you own.
• Standardised measurement. HarmBench (Mazeika et al., 2024) fixes behaviours, judges and budgets so attack methods and defences can be compared. They compared 18 attack methods on 33 target models and defences.

ASR(category) = #goals where ≥1 attack within budget B succeeds / #goals
success = judge says the target failed  AND  the attack kept the test goal's intent

ASR is only meaningful with its budget, judge and goal set stated. The same attacker can show 10% at B = 5 queries and 70% at B = 100.

LLM-as-a-judge calibration

Judges mark open-ended answers, and in red teaming they decide whether an attack "worked". Zheng et al. (2023) found strong judges agreed with human preferences over 80% of the time, about the level at which humans agree with each other. They also documented position bias (preferring the first or second answer), verbosity bias (preferring longer answers) and self-enhancement bias (preferring answers from the same model). A practical calibration routine:

1. Hand-label 100–300 examples, stratified by category. Keep a held-out set.
2. Measure agreement with humans: precision and recall on "failure", plus Cohen's κ. Use Module 01's cross-validated threshold tuning to pick a cut-off that meets minimum precision and recall.
3. Check the score itself with a reliability diagram and Platt scaling (Module 01, calibration) if you'll use it as a probability.
4. Run bias probes: swap answer order, pad an answer with filler, and swap which model wrote it. A calibrated judge's verdict shouldn't move.
5. Re-check on attack outputs. Adversarial text is out of distribution for the judge too, so a judge tuned on normal traffic can be fooled by the same tricks as the target.

3. The practice project

(a) Laptop version: red-team your own contact-details filter

Use a deliberately harmless policy so the whole loop is safe to run and share: a classifier that should flag messages containing contact details (an email address or phone number), for example a filter in a marketplace chat. Use made-up data only (example.com, 555 numbers).

1. Target: start with a regex, then a small fine-tuned classifier (a DistilBERT-sized model trains in minutes on CPU or Colab).
2. Seed goals: 100+ messages that must be flagged, plus 200+ benign messages that mustn't (to track false positives).
3. Attack categories: formatting (spacing, separators), spelling-out (" at ", " dot "), character substitution (look-alike Unicode), context wrapping (burying it in a long message), and splitting across turns. Each mutation must keep the details readable to a human. That's the "intent preserved" check.
4. Tree search: from each seed, branch width mutations, score with the target, keep the lowest-scoring children, go depth levels. Count queries.
5. Report: ASR per category at a fixed budget, queries to first success, and the false-positive rate on the benign set.
6. Close the loop: add successful attacks to training, retrain, then measure ASR on a held-out set of attacks from a fresh search. Freeze old successes as a regression suite.
7. Upgrade the attacker: swap the hand-written mutators for a small local instruct model that proposes rewrites (TAP-style), and add a judge that checks the rewrite still contains the details. Calibrate that judge on 100 hand-labelled rewrites first.

import random, re

def detector(t):  # stand-in for YOUR classifier: P(contains contact details)
    return 1.0 if re.search(r"[\w.]+@[\w-]+\.\w+|\d{3}[- ]?\d{3}[- ]?\d{4}", t) else 0.0

MUT = {  # each keeps the details readable to a human
    "spell-out": lambda t: t.replace("@", " at ").replace(".com", " dot com"),
    "spacing":   lambda t: re.sub(r"(?<=\d)(?=\d)", " ", t),
    "fullwidth": lambda t: t.translate({ord(c): ord(c) + 0xFEE0 for c in "0123456789@"}),
    "wrapper":   lambda t: f"reminder for later, {t}, thanks",
}

def search(seed, depth=3, width=2, rng=None):
    frontier, q = [(seed, [])], 0
    for _ in range(depth):
        kids = []
        for t, path in frontier:
            for name in rng.sample(sorted(MUT), width):
                c = MUT[name](t); q += 1
                s = detector(c)
                if s < 0.5:
                    return path + [name], q
                kids.append((s, c, path + [name]))
        kids.sort(key=lambda k: k[0])
        frontier = [(c, p) for _, c, p in kids[:width]]
    return None, q

rng = random.Random(0)
seeds = [f"email me at user{i}@example.com" for i in range(50)] + \
        [f"call me on 555-{100 + i}-{1000 + i}" for i in range(50)]
hits = [search(s, rng=rng) for s in seeds]
print(f"ASR {sum(p is not None for p, _ in hits) / len(seeds):.0%}")

Expected outcome: the regex falls to almost every seed (this exact script prints 97%), which is the point: it shows the harness works. The interesting numbers come once you swap in a trained classifier and a model-driven attacker: per-category ASR before and after one round of adversarial training, measured on held-out attacks.

(b) Full version from the post

An attacker LLM running TAP-style tree search against your own safety classifier (for example a guard model you fine-tuned), inside a sandbox. Use a written policy with categories, a goal set per category, a calibrated judge, a fixed query budget, and a non-tree baseline at the same budget so you can show the tree search earns its cost. Frameworks help: PyRIT orchestrates attacker, target and scorer; garak runs libraries of known probes; Inspect makes the whole thing a reproducible eval. Cost: each query involves the attacker, the target and usually the judge. 200 goals × 15 target queries each (width 3, depth 5, keeping one branch per level) is 3,000 target queries per run, wider trees multiply that, and attacker and judge calls come on top, so use small open models locally or budget API spend before you start. A 7–8B attacker runs on one 24 GB GPU.

(c) Pitfalls

• Judge fooled with the target. If the judge shares the target's blind spots, ASR is wrong in both directions. Audit a sample of "successes" and "failures" by hand.
• Intent drift. Attacks that succeed by dropping the test goal aren't successes. Check the goal is still there.
• Training on the test. Retraining on found attacks then re-scoring the same attacks measures memory. Use a fresh search for the after-number.
• Ignoring false positives. A classifier that flags everything has 0% ASR and is useless. Always report the benign-set false-positive rate.
• Budget-free ASR. Always state the query budget and number of goals, with a confidence interval (100 goals gives roughly ±10 points at 50%).

4. Reality check

5. In this course

• Module 01 · Decision-model evals: thresholds, cross-validated tuning and calibration, the toolkit for your judge and your classifier.
• Module 04 · Evaluation that doesn't fool you: splits by session, cutoff dates, paired bootstrap, slices.
• Module 07 · Preference alignment: reward hacking, the training-time version of a model gaming its metric.
• 30u30 Day 30 · RL from Human Preferences: where learned judges (reward models) come from.
• Stage 04 (verifier filtering) and Stage 05 (reward models): the same judge-reliability questions, used for training instead of testing.

6. Why it matters for robotics

7. Self-check

8. Courses & content

Start with the free Hugging Face Evaluation Guidebook for contamination and judges, then build the harness in Inspect. Add PyRIT when you get to the red-teaming loop.

Full course map for all 12 stages →

9. Papers & docs

• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Perez et al. (2022). Red Teaming Language Models with Language Models
• Mehrotra et al. (2023). Tree of Attacks: Jailbreaking Black-Box LLMs Automatically (TAP)
• Chao et al. (2023). Jailbreaking Black Box Large Language Models in Twenty Queries (PAIR)
• Zou et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models (GCG)
• Mazeika et al. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• Oren, Meister, Chatterji, Ladhak & Hashimoto (2023). Proving Test Set Contamination in Black Box Language Models
• Zhang et al. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k)
• White et al. (2024). LiveBench: A Challenging, Contamination-Limited LLM Benchmark
• Brown et al. (2020). Language Models are Few-Shot Learners (13-gram contamination analysis)
• Docs: Inspect · PyRIT · garak