Skip to content

Module 23. Evaluating agents

After this module you will be able to

  • Explain eval as an automatic test of an agent: a task, a reference, a score.
  • Assemble a golden set and say why the reference must not leak into tuning.
  • Tell a regression from general progress and explain why the average score hides it.
  • Name the ways to score an answer and the pitfall of each, including a model-as-judge.
  • Close Part VI: the agent, retrieval with citation checking, and evaluation into one development loop.

Time: about one week. Prerequisites: Module 21, Module 22 and Module 1. Notebook: open in Colab · notebooks/23-evaluating-agents.ipynb

Why this

You have built an agent (Module 21) and taught it to answer from documents with citation checking (Module 22). One question remains, and without it this is not engineering but hope: how do you know it works — and that it did not break after an edit?

"I tried a couple of queries, it seems to answer" survives neither a version change nor a second developer. You need an eval — an automatic test of the agent: a task with a known good answer, a run, a score. Exactly what Module 1 did with a number from a paper, only now the claim under test is "the agent solves this task".

task agent answer check pass fail reference

The golden set

One test guarantees nothing: the agent might answer it by luck and fail everything else. You need a golden set — a set of tasks with reference answers, curated by hand and frozen. It plays the role of the baseline from Module 1 and the holdout from Module 7: it is the anchor you compare against.

And it carries the same iron rule as the holdout: the reference must not leak into tuning. The moment you start twisting the agent until the golden set turns green, you are optimizing for the test, not the task — and the number rises while quality does not. This is Goodhart from Module 16: a measure that becomes a target stops being a measure.

An answer can be scored in several ways, and each has its price:

  • Exact match — cheap and strict, but fails a correct answer over one extra space.
  • Key match (a number, a fact) — more forgiving; the citation check from Module 22 is exactly this case.
  • A model-as-judge — scores a free-form answer with another model. Powerful and dangerous: the judge can be talked round by length and a confident tone, and that is Goodhart again. The judge itself has to be checked on a golden set.

The regression hides in the average

Now the main point of this module. You ship a new version of the agent and run it on the same golden set. The average score went up — progress, it would seem. But the aggregate folds a fix and a break into one number and hides the regression: a task that used to be solved and now is not.

v1 v2 task 1 task 2 fixed task 3 broke — regression average 2/3 2/3 the same — the regression is invisible

A regression is a per-task diff, not a diff of averages. It is the same move as the trace from Module 21: to find the break you compare not total with total, but step with step. Here — task with task.

Press "run v2". The average rises from 4/6 to 5/6 — two tasks were fixed. But one passing task turned red: v2 started looping on a tool error. The average score does not show it; the per-task diff shows it at once. That is why eval is measured not as one number but as a table: what was fixed, what broke.

Evaluating an agent is Module 1 for a program. In Module 1 you checked a claim from a paper: baseline, noise, repetition. Here the claim under test is "the agent solves the task", the golden set is the baseline and the holdout (Module 7) at once, and checking the answer against the reference is the citation check from Module 22. And the same trap: the moment eval becomes the target of tuning, Goodhart from Module 16 kicks in, and the diff of runs is read with the same eye as the diff of traces from Module 21.

Practice

Part 1. The notebook

Open notebooks/23-evaluating-agents.ipynb. Only numpy and matplotlib.

What is inside:

  1. Golden set and score: we assemble a set of tasks, run the agent, compute pass@1. The claim "it works" becomes a number you can rerun.
  2. Regression: two runs, a per-task diff. The average rose — and one task broke. We show that the aggregate hides it.
  3. Goodhart of the eval: optimize the metric directly — the score rises, quality does not. And pass@k versus pass@1: more attempts inflate the apparent success.

Part 2. Your own golden set

Take any task you solve with an agent or a model.

  1. Assemble 10 tasks with reference answers. Which are honestly hard, and which did you pick because the agent already solves them?
  2. How will you score an answer: exact match, by key, a judge? Where will the judge be wrong?
  3. Make an edit, run twice and find the per-task diff. Is there a regression?
  4. What happens to the score if you start twisting the agent straight against this golden set?

Assignment

  1. Implement an eval: a function takes a task and a reference, runs the agent, returns pass/fail. Compute pass@1 over the golden set.
  2. Make two runs of the agent and compute the per-task diff: fixed, broken, unchanged. Find a case where the average rose but a regression is present.
  3. Add a model-as-judge (a stub) and show how it is fooled by answer length while the substance is unchanged.
  4. Build pass@k for k = 1, 2, 5 and show that growing k inflates the apparent success without any gain in quality.
  5. Take the golden set into tuning: twist the agent against it and show how the score on it diverges from the score on a held-out set.

Self-check

  1. What is an eval and which claim does it test?
  2. Why a golden set, and why must the reference not leak into tuning?
  3. Name three ways to score an answer and the weak spot of each.
  4. Why does the average score hide a regression, and how do you see it?
  5. Which module does "the reference must not leak into tuning" rhyme with?
  6. Why is pass@k at a large k misleading?
  7. Where else in the course did step-by-step diff appear instead of a diff of totals?

Next

This closes Part VI: an agent with a readable trace (Module 21), retrieval with citation checking (Module 22) and an evaluation that catches regressions — this is a development loop, not a one-off run. In Part VII the course reaches the frontier: how to read a paper, how the field and its leaderboards are organized, how to reproduce a result and how to keep up. The evaluation from here is exactly the tool with which you read other people's claims.

Eval turns "seems to work" into a number you can rerun; a golden set is the baseline and the holdout at once; a regression lives in the per-task diff, not in the average score. Evaluation is not a report at the end but a test that catches the break before the user does.

Principle

The average score folds a fix and a break into one number and hides the regression. Development stands not on the metric having risen, but on your knowing which task, exactly, broke.