Evaluating AI agents

A companion guide, "AI agents for web apps," builds the thing this one measures. It writes the agent loop twice — first by hand against the raw Anthropic SDK, until the while loop underneath every framework is something you could type from memory, then again under Pydantic AI, where the loop disappears and the parts that remain are the ones that matter — and wires the result into a FastAPI backend streamed to a Next.js front end. The running example is PlanurLife, a planning assistant that reads one signed-in user's private goals, tasks and journal and writes new rows into them: add a task to a goal, move some sessions to mornings, answer a question about the week. Read that guide first; this one assumes the agent exists and runs. What it is for is the question that comes next. You change a prompt, the pass rate moves from 24/30 to 26/30, and you need to know whether the change actually helped. That turns out to be a statistical question, and most of this guide is the statistics, built from the ground up, that separate a real improvement from noise.

The agent is a distribution, not a function

Send the same message through the same agent five times and you can get five different runs. Once it searches the user's goals and then creates the task; once it creates the task first and searches after; once it asks a clarifying question and stops. Nothing changed between the runs — not the prompt, not the model, not one row in the database. The model draws its next token from a probability distribution, and that is not a setting you can switch off.

So "correct" is not a property your agent has on a given input. It is a probability — the chance that this input produces an acceptable run — and every number in this guide estimates one.

Turning the temperature down does not rescue you. Thinking Machines Lab sampled 1,000 completions at temperature 0 from one model and got "80 unique completions, with the most common of these occuring 78 times," the outputs first diverging "at the 103rd token"; only with special batch-invariant kernels were "all of our 1000 completions … identical." The cause is floating-point arithmetic whose result depends on what else the server batched alongside your request, which is nothing you control.

Measured on agents instead of raw completions, Bjarnason, Silva and Monperrus ran 500 SWE-Bench-Verified tasks across 10 runs, 3 models and 2 scaffolds. Single-run pass@1 scores — the one-attempt success rate, whose whole family a later chapter takes apart — "vary by 2.2 to 6.0 percentage points depending on which run is selected, with standard deviations exceeding 1.5 percentage points even at temperature 0," and "temperature 0 does not result in determinism."

Three things you meet later are consequences of that one fact, and each is easier to accept as a consequence than to memorise as a convention:

Partial credit exists because a single run is one sample from a distribution, so a case is rarely purely right or wrong.
Trials exist because one sample is a noisy estimate of the thing you actually want to know.
Judges exist because the space of acceptable outputs is too large to write down, so you cannot enumerate the right answer and compare against it.

The plan this guide follows is three moves in order: ship the assistant, build a harness that scores it, then feed real user reactions back into that harness. The order is not arbitrary, and the next chapters earn it.

Every good testing instinct you have misfires here

What follows is an argument, not a finding: no study sits behind this table, only the consequences of the section above it.

Slow down here, because your instincts about tests are correct and every one of them is wrong about evals.

The instinct	Why it's right for tests	Why it's wrong for evals
A flaky test is a broken test	Non-determinism in a unit test is a bug in the test	Non-determinism is the subject matter. A case that passes 8 times in 10 is not flaky, it is informative. Deleting it deletes your signal.
Green means ship	A test is a proof over the cases it covers	Green over 30 samples of a distribution is a point estimate with a wide interval. Green means "not obviously broken."
Assert on the exact output	Determinism makes exact assertion cheapest	Exact-match on prose tests phrasing. Assert on the trajectory and the side effects instead.
One assertion per test	Isolates the failure	One assertion per case throws away most of the information in an expensive run. The run already happened — attach many evaluators to it.
Tests are code, in the repo	Right	Cases are data. Users create them, not you, and the format has to survive being written by a UI.
Mock the dependency	Isolation	Mocking the model turns the eval into a test of your plumbing. `TestModel` and `FunctionModel` are for testing the harness, never for evaluating the agent.
100% coverage is the goal	Coverage is measurable and monotone	There is no denominator. The input space is natural language.

You cannot write the rubric first

The last instinct is to specify what good looks like before measuring anything. That one fails for a reason with a name.

Shankar and colleagues studied people building LLM evaluations and named it criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria." They found "some criteria appears dependent on the specific LLM outputs observed (rather than independent criteria that can be defined a priori)."

The circularity is real and not a personal failing. The way out is to grade about thirty real outputs before writing any rubric, and let the criteria fall out of what actually annoyed you. Ship first, measure second, for the same reason: the planner has to disappoint you before a harness is worth building, or you encode your guesses about failure instead of your observations of it.

A trace before a score

Observability is not the reward for having evals. It is a prerequisite, and in this stack that is a runtime fact, not an argument.

Pydantic Evals' trajectory evaluators — the ones that check which tools were called, in what order, how many times — read the OpenTelemetry span tree. OpenTelemetry (OTel) is the vendor-neutral standard for emitting traces, the timed record of what happened during a run; a span is one operation in that record, a single model request or one tool call, with a start, an end and a bag of attributes. With no instrumentation configured, there is no span tree, and the evaluators do not silently pass:

Ignore the names in the left-hand column for now — the chapter "Assertions before judges" builds that set. All that matters here is the ✗.

text · the same eval, with the two logfire lines deleted

                   Evaluation Summary: no_instrumentation
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Case ID                   ┃ Assertions                                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ add_task_to_existing_goal │ ToolCorrectness: ✗                           │
│                           │   Reason: No span tree available — ensure    │
│                           │ logfire/instrumentation is configured.       │
│                           │                                              │
│                           │ MaxToolCalls: ✗                              │
│                           │   Reason: No span tree available — ensure    │
│                           │ logfire/instrumentation is configured.       │
├───────────────────────────┼──────────────────────────────────────────────┤
│ Averages                  │ 0.0% ✔                                       │
└───────────────────────────┴──────────────────────────────────────────────┘

The agent ran correctly. The evaluators had nothing to look at. Most test tooling skips a check it cannot run; these fail it, so tracing is a hard dependency and not an advanced topic you get to later.

Two calls fix it, and they are the same two calls that give you production tracing. Logfire is Pydantic's own OpenTelemetry backend, hosted — which is a decision, not a default, and the next chapter is about what you must not send it:

python · instrument once, at startup

import logfire
logfire.configure()
logfire.instrument_pydantic_ai()

What a turn looks like as a tree

Here is a real span tree from a two-tool agent run — one user message, two tool calls, three model requests:

text · span name and kind, exported to an in-memory collector

- invoke_agent planner   [INTERNAL]
  - chat function:script:   [CLIENT]
  - execute_tool search_goals   [INTERNAL]
  - chat function:script:   [CLIENT]
  - execute_tool create_task   [INTERNAL]
  - chat function:script:   [CLIENT]

Two things to read off it. It is flat, not nested: the tool spans are siblings of the chat spans, not children, because in the loop they alternate instead of containing one another. And three model requests for two tool calls is the arithmetic you already know from the loop, now visible as a shape you can query.

The names are not Pydantic's invention. invoke_agent, execute_tool and chat are OpenTelemetry's GenAI semantic conventions, which also define create_agent, invoke_workflow and a plan span.

Their status is worth knowing before you build a dashboard on them. Every gen_ai.* attribute in the registry is badged Development; the conventions have moved out of the main semantic-conventions repository into a dedicated one; and that repository has no tagged releases. You can see it in the installed package: the 60-odd gen_ai.* constants live only in opentelemetry.semconv._incubating, and importing the stable path for them raises ImportError. The only Stable badges on a GenAI span belong to borrowed general attributes such as error.type and server.port. Nothing gen_ai.* is stable.

The fields that make a failure debuggable a week later

Instrumentation gives you a lot for free. Off the parent invoke_agent span and its children you get gen_ai.operation.name, gen_ai.agent.name, gen_ai.conversation.id, gen_ai.request.model and gen_ai.response.model, gen_ai.provider.name, per-request gen_ai.usage.input_tokens and output_tokens, a Pydantic-specific gen_ai.aggregated_usage.* roll-up on the parent that exists so backends do not double-count, plus gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.definitions, span status, durations and the standard trace and span ids.

What is missing is everything that ties a turn to your system, and you attach it yourself:

A pseudonymous user_id. Not the email.
The prompt version. The conventions give you gen_ai.prompt.name for the name of a prompt (registry example: analyze-code). For the version there is no standard attribute in the package as shipped — semantic-conventions 0.65b0 has gen_ai.prompt.name and nothing for a version — so attach your own, a prompt.version on the span. Without it, a trace from six weeks ago describes behaviour you may already have fixed and you cannot tell.
The app's commit SHA, and the route.
The ids of every row the turn wrote. This is the field people forget, and it is the one that makes a week-old failure debuggable. Without it you can see that create_task was called; you cannot see what it created, whether the goal it attached to exists, or whether the user deleted it four minutes later.

Do not sample yet

The reflex once tracing is on is to sample it down. OpenTelemetry's own guidance opens its "when not to sample" list with the case you are in: "You generate very little data (tens of small traces per second or lower)," adding that sampling "might introduce other unexpected costs if not performed well. It could be cheaper to allocate more resources for observability instead." Its "when to sample" list opens at "1000 or more traces per second."

At an open beta's traffic the answer is to trace every turn, and to notice that sampling away failures you have not seen yet is exactly the wrong trade while you are still discovering what goes wrong.

When it does become relevant, the axis that matters is your backend's billing unit, not its brand. Logfire and Langfuse bill per event — a span, a log, a score. Braintrust and W&B Weave bill per byte ingested. An agent app produces few events carrying large payloads, which is the cheapest possible shape for per-event billing and the most expensive for per-byte. Logfire's Personal tier covers 10 million records a month and is hard-capped there, so it cannot bill you at all, and eval scores are metered at $0. It is also not a one-way door: the Logfire SDK can be pointed at any OTLP endpoint with send_to_logfire=False.

What your traces must not contain

You have just turned on something that records, by default, every word your users write.

The OpenTelemetry specification is explicit about this. Under Capturing instructions, inputs, and outputs, gen-ai-spans.md says model instructions, user messages and model outputs "are considered sensitive and are often large in size," that recording them "may be problematic due to high storage costs, regulatory requirements, or the need to enforce different access models," and then, normatively:

OpenTelemetry instrumentations SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in.

It names three patterns in order — don't record; record on the span, "best suited … for example, in pre-production environments"; or store content externally and record references, the one "recommended in production environments." Every content attribute is requirement-level Opt-In and carries a warning that it "is likely to contain sensitive information including user/PII data."

Pydantic AI 2.36.0 chooses otherwise. In models/instrumented.py:

python · the default that decides what leaves your server

class InstrumentationSettings:
    include_binary_content: bool = True
    include_content: bool = True          # prompts, completions, tool args and results
    include_model_request_parameters: bool = True

With Logfire configured and no further action, an app holding private journals ships those journals to a third-party service. That is a defensible default for a developer tool and the wrong one here.

The scrubber is a credentials filter

The natural next thought is that Logfire's redaction will catch it. It will not, and it is worth seeing the measurement instead of taking the claim. Running the real Scrubber with only its built-in patterns over a dict of journal-shaped content:

text · the default scrubber against private content and secrets

journal_entry          UNCHANGED  "Felt awful today. Therapy helped a bit but I'm still not sleeping."
journal_entry_2        REDACTED   "[Scrubbed due to 'session']"
goal_title             UNCHANGED  'Leave my job by March and move closer to Dad'
health_note            UNCHANGED  'Increase sertraline to 100mg, ask GP on Thursday'
user_email             UNCHANGED  '[email protected]'
db_password            REDACTED   "[Scrubbed due to 'password']"
api_key                REDACTED   "[Scrubbed due to 'api_key']"

Every credential was caught and no private content was: the pattern list is password, secret, api_key, session, cookie, jwt, csrf and a dozen more of the same kind. It has no concept of "this is a journal."

And the one entry it did catch, it caught for the wrong reason. "Second session with the new therapist" matched the pattern meant for session tokens, and the whole sentence was replaced. So the scrubber both under-protects private content and, at random, silently destroys it.

And it is switched off for the fields that matter

That is the smaller finding. The larger one is that scrubbing is deliberately disabled for the LLM message attributes. BaseScrubber.SAFE_KEYS lists gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and pydantic_ai.all_messages, and the dispatch is a straight passthrough with no recursion:

text · the same secret string, in two different attributes

=== is the attribute on SAFE_KEYS? ===
  gen_ai.input.messages            SAFE (never scrubbed)
  gen_ai.output.messages           SAFE (never scrubbed)
  pydantic_ai.all_messages         SAFE (never scrubbed)
  gen_ai.tool.call.arguments       scrubbed normally
  planurlife.journal               scrubbed normally

=== "My password is hunter2 and the therapy session helped." ===
  under gen_ai.input.messages      -> UNCHANGED   (scrub notes: [])
  under gen_ai.tool.call.arguments -> "[Scrubbed due to 'password']"

The literal word "password," unredacted, in the one attribute you would most expect to be filtered. This is intentional and documented, and the stated reasoning is sound: LLM conversation contains words like "password" and "secret" innocently all the time (false positives), and models emit sensitive data without using any keyword a regex could catch (ineffective detection). Pydantic's own conclusion is the one to act on: "if your LLM interactions might contain sensitive data, the recommended approach is to exclude message content from logging entirely rather than relying on scrubbing."

What you keep when you turn content off

logfire.instrument_pydantic_ai(include_content=False) is not a lobotomy. Running the same agent both ways, the shape survives and the text goes:

text · the same two spans with include_content=False

→ chat function:script:
   gen_ai.input.messages = [{"role":"user","parts":[{"type":"text"}]}]
   gen_ai.output.messages = [{"role":"assistant","parts":[{"type":"tool_call",
                              "id":"pyd_ai_c8a7…","name":"search_goals"}]}]
   gen_ai.usage.input_tokens = 58
   gen_ai.usage.output_tokens = 5

→ execute_tool search_goals
   gen_ai.tool.call.id = pyd_ai_c8a7a4c397e946a6b3a714b40a8b2ae7
   gen_ai.tool.name = search_goals
   # gen_ai.tool.call.arguments and gen_ai.tool.call.result are GONE

You keep message roles, part types, which tools were called with which call ids, token counts, timings, span status — the entire trajectory. You lose every word the user wrote and every word the model wrote back.

That is still a fully debuggable trace for a tool-calling agent, and the trajectory evaluators of the next chapter read tool names and call structure, not content, so ToolCorrectness, TrajectoryMatch, MaxToolCalls and MaxModelRequests all keep working. ArgumentCorrectness does not, because it reads gen_ai.tool.call.arguments. That one evaluator is the price.

So: content on in development, off in production, and argument-level checking from your own evaluators reading your own database, not from the trace. If you ever genuinely need content in production telemetry, the documented route is an OTel Collector in between with redaction and transform processors — real infrastructure, not a flag. For an application whose next planned phase is per-user envelope encryption, treating this as a config toggle would be strange: the trace is the one copy of the journal that the encryption does not cover.

Assertions before judges

"LLM-as-a-judge" is the headline, and reaching for it first is the most expensive mistake available here. Almost everything that actually goes wrong in a tool-calling agent is checkable with ==:

it called the wrong tool, or called nothing at all;
it skipped a tool it needed — searched nothing, then wrote;
it called one tool six times when once was asked for;
it passed an id that does not exist;
it wrote a row belonging to a different user.

None of those needs a model to detect. Reach for a judge only for the residue — the parts genuinely about language.

The evaluators in the box

pydantic_evals 2.36.0 ships two families. The plain ones compare values: Equals, EqualsExpected, Contains, IsInstance, MaxDuration, HasMatchingSpan. The interesting ones read the span tree you turned on earlier and check the trajectory:

python · the agentic evaluators, with their real constructor signatures

ToolCorrectness(expected_tools: list[str], allow_extra, include_failed, evaluation_name)
TrajectoryMatch(expected_trajectory: list[str], order, include_failed, evaluation_name)
ArgumentCorrectness(tool_name, expected_arguments: dict, match_mode, occurrence, ...)
MaxToolCalls(max_calls, include_failed, evaluation_name)
MaxModelRequests(max_requests, evaluation_name)

Their own module docstring makes the case: "They are deterministic and require no LLM calls, so they are cheap to run and produce the same score for the same trace." One caveat from the same docstring — they "only see tools whose execution produces a local span," so a provider-side tool such as a server-run web search is invisible to them.

Two behaviours the flat list hides, and both bite. MaxToolCalls counts failed attempts against the budget by default (include_failed=True), the only one of the five that does; the other four default to False. And TrajectoryMatch does not return a pass/fail tick at all — it returns a float, the F1 overlap between the actual and expected tool sequences, so it is averaged like a score rather than counted like an assertion. The rest return a plain bool.

A dataset is a list of cases and a list of evaluators

python · one case, five evaluators, two of them yours

dataset = Dataset[str, str, dict](
    name='planner_v1',
    cases=[Case(
        name='add_task_to_existing_goal',
        inputs='Add a 3km run on Tuesday to my running goal',
        expected_output='task created',
        evaluators=[
            ToolCorrectness(expected_tools=['search_goals', 'create_task']),
            MaxToolCalls(max_calls=2),
        ],
    )],
    evaluators=[IsInstance(type_name='str'), WroteOnlyForUser(), GoalIdIsReal()],
)

Case-level evaluators run for that case; dataset-level ones run for every case. WroteOnlyForUser and GoalIdIsReal are the two that matter, because no library can write them:

python · domain evaluators — the tenant boundary and referential integrity

@dataclass
class WroteOnlyForUser(Evaluator[str, str]):
    """Every row written during the case belongs to the case's user."""
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:
        rows = ctx.attributes.get('written', [])
        return all(r['user_id'] == 'user-1' for r in rows)


@dataclass
class GoalIdIsReal(Evaluator[str, str]):
    KNOWN = {'goal-run-a-5k'}
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:
        rows = ctx.attributes.get('written', [])
        return all(r['goal_id'] in self.KNOWN for r in rows)

Side effects reach an evaluator through set_eval_attribute('written', [...]), called inside the task, which lands on ctx.attributes. Do not reach for ctx.metadata — that is the case's declared metadata, a different thing, and getting it wrong fails the run, not the assertion. For a database-backed harness, Dataset.evaluate(..., lifecycle=...) gives you setup() / prepare_context() / teardown() per case; cases run concurrently by default, so each needs its own user_id, not a shared fixture.

What the report says, and what it does not

Run two prompt versions — a careful one and a sloppy one that skips the search, invents a goal id and creates two tasks. Turn on include_reasons=True, because the bare ✔/✗ strip is not actionable:

text · the sloppy prompt's assertions column

│ ToolCorrectness: ✗                                                  │
│   Reason: missing tools: 'search_goals' (x1);                       │
│           unexpected tools: 'create_task' (x1)                      │
│                                                                     │
│ MaxToolCalls: ✔                                                     │
│   Reason: 2 tool call(s), budget=2                                  │
│                                                                     │
│ IsInstance: ✔                                                       │
│ WroteOnlyForUser: ✔                                                 │
│ GoalIdIsReal: ✗                                                     │

Read that closely, because it teaches three things at once and the middle one is the most useful.

ToolCorrectness produced a real diagnostic — which tool was missing, which was unexpected, how many times — from the span tree alone, with no model call. That is the difference between a failing eval and a debuggable one.

MaxToolCalls passed, and it should not have. The bug in the sloppy prompt is that it created two duplicate tasks. The budget was set to 2. Two calls is within a budget of two, so the assertion written to catch duplicate task creation reported a pass, and nothing in the report says so. An eval checks exactly what you specified, and you will get the threshold wrong the first time. The green tick is not evidence that the behaviour is right; it is evidence that the behaviour matched a number you chose before you had seen the failure.

The tenant check passed and referential integrity failed. The agent wrote rows for the right user, attached to a goal that does not exist. No library evaluator finds that, and no text-matching evaluator would either, because the reply — "Done!" — is unobjectionable. The bug is only visible in the database.

Assertion, score, scorer: the words are not shared

The vocabulary genuinely differs between tools, and learning a fake unified glossary will cost you later.

Concept	Pydantic Evals	Braintrust	LangSmith
One input + expected output	`Case`	a row in a `Dataset`	an `Example`
The grading unit	`Evaluator`	`scorer`	`evaluator`
Boolean verdict	`assertion`	a scorer returning 0/1	a boolean feedback key
Numeric verdict	`score`	scorer return value in [0,1]	a numeric feedback key
Repeat of a case	`repeat=N`	—	—

The distinction worth internalising is Pydantic's, because it changes behaviour and not just vocabulary: an evaluator returning bool produces an assertion, which gets a ✔/✗ and contributes to a pass rate; returning float or int produces a score, which gets averaged; returning str produces a label; returning a dict produces several named results from one evaluator. "Trial" is not a Pydantic word at all: the parameter is repeat.

The gate

EvaluationReport is not print-only. report.averages() gives you a plain object, and report.trace_id links the run back to its own trace:

text · the whole CI gate, from averages().assertions

GOOD PROMPT     assertions=1.0  metrics={'requests': 3.0, ...}  trace_id=01a02de67a53…
SLOPPY PROMPT   assertions=0.6  metrics={'requests': 3.0, ...}  trace_id=01a02de67a77…

gate: assertion pass rate must be >= 0.9
  good   -> PASS (1.00)
  sloppy -> FAIL (0.60)

Gate on a floor, not on a delta. The next two chapters are about why a delta at this sample size is not a measurement, while a collapse still is — and they start one level down, at what the number 80% even is.

What a pass rate is

At the end of "Describing a pile of numbers" — a chapter in the companion mathematics guide "Shape and change" — there is a parenthesis you were asked to take on faith. Working out a standard deviation, you divide by how many numbers you have. Except, it said, when you are using a sample to estimate a larger population's spread, where you divide by n − 1 instead, "for reasons that belong to a statistics course." This chapter is the part of that course a pass rate needs, and the word carrying it is estimate.

Here is the number in question. Your eval set has 30 cases; the agent passes 24; you write 80%. You have been reading that as the agent's score, a fact about the agent the way its parameter count is a fact. It is not. It is a measurement, and it has the property every measurement has: run it again and it moves.

A statistic is a guess at a parameter

Think of the agent, on this kind of case, as a bent coin. The coin has some true probability of landing heads — call it p — and every time you run a case you flip it once. You cannot read p off the coin. It is a fact about the coin's construction, fixed but hidden. All you can do is flip and count.

Flip 30 times, get 24 heads, and 24/30 = 0.80 is your sample proportion, written p̂ and read "p-hat." The hat means estimated: p̂ is what you measured, p is what you wanted. p̂ is a statistic — a number computed from a sample. p is a parameter — a number describing the whole population the sample came from, here the endless population of runs you could have made and didn't.

This is the one genuinely new idea in the chapter, and everything else falls out of it. Your 30 cases are a sample; the agent's true behaviour is the population; 80% is a statistic estimating a parameter you will never see directly. Once "80%" stops being a fact about the agent and becomes a guess with an error attached, the rest is machinery.

One case is a coin flip; the run is the count

Give the pieces their names, because the machinery is named after them. A single case is a Bernoulli trial: one event with two outcomes, pass or fail, probability p of the one you are counting. Thirty independent Bernoulli trials, and the number of passes X is a binomial count — "binomial" meaning "two names," pass and fail. They are independent, so the multiplication rule from "Chance: counting the ways" applies: the probability of any one particular sequence of passes and fails is a product of p's and (1 − p)'s. And p̂ = X/n is that count turned back into a rate.

How much a single flip varies

You already have "the average of the squared distances from the mean" as the recipe for spread, one square root short of the standard deviation. That quantity has a name this chapter can use: the variance. Standard deviation is its square root; variance is the step before it.

What is the variance of a single pass/fail? Code a pass as 1 and a fail as 0. The mean of that one flip is p, because it is 1 with probability p. Its two possible distances from the mean are (1 − p) when it lands on 1, and (0 − p) = −p when it lands on 0. Square each, weight each by how often it happens, and add:

text · the variance of one pass/fail outcome

variance = p·(1 − p)²  +  (1 − p)·p²
         = p(1 − p)·[(1 − p) + p]
         = p(1 − p)

So one flip has variance p(1 − p). Look at what that says. It is largest at p = ½ — an outcome you cannot predict varies the most — and it shrinks toward zero as p nears 0 or 1:

text · run it: variance of one flip at four rates

p = 0.50   →   p(1−p) = 0.2500
p = 0.80   →            0.1600
p = 0.95   →            0.0475
p = 0.99   →            0.0099

A near-perfect agent barely varies from run to run. That is worth holding onto: a high pass rate is easier to pin down than a middling one, and the next formula says by how much.

Re-run the whole eval, and the rate itself has a distribution

Here is the move "Shape and change" never makes. Imagine re-running your entire 30-case eval many times over — a fresh sample of the agent each time. Each run hands you a slightly different p̂: 0.80, then 0.77, then 0.83, then 0.80 again. Those p̂ values are not the data. They are a distribution of the measurement itself, the sampling distribution, centred on the true p, and its spread is the whole of your uncertainty.

You do not have to imagine it. Simulate 200,000 re-runs of a 30-case eval on an agent whose true rate is exactly 80%, and look at the p̂ values that come back:

text · 200,000 simulated re-runs of a 30-case eval, true p = 0.80

mean of 200,000 p-hats     0.7999      (true p = 0.80)
sd of those p-hats         0.0731

The re-run rates pile up around 0.80, the true value, with a standard deviation of about 0.073 — 7.3 percentage points. That scatter is unavoidable, and it is what "80% from 30 cases" is actually worth.

Standard error: the spread of the estimate

That 0.073 has a name and a formula. The standard deviation of the sampling distribution — the typical distance of p̂ from the true p — is the standard error (SE). It is not the standard deviation of your data; it is the standard deviation of your guess. Keep the two apart, because the words look alike and mean opposite-facing things: the data's sd asks "how spread out are the cases," the standard error asks "how spread out is the number I am reporting."

You can derive it from what you already have. The variance of one flip is p(1 − p). p̂ is the average of n independent flips, and averaging n independent things divides their variance by n — so p̂ has variance p(1 − p)/n, and the standard error is its square root:

text · the standard error of a proportion

SE = √( p(1 − p) / n )

Check it against the simulation. At p = 0.8 and n = 30, √(0.16 / 30) = √0.005333 = 0.0730. The formula and the 200,000-run experiment agree to three decimals, which is the point: the formula is not a rule to trust, it is a shortcut for the experiment you could always run.

The shape of that formula is the practical lesson. n sits under the square root, so the error falls like 1/√n — to halve your uncertainty you must quadruple your sample:

text · SE at p = 0.8, as n quadruples

n =  30   →   SE = 0.0730
n = 120   →   SE = 0.0365
n = 480   →   SE = 0.0183

Four times the cases for half the error is the exchange rate you are stuck with, and it is why a precise eval set is expensive.

The bell comes back, over estimates this time

One more borrowed fact and the picture closes. "Shape and change" drew the normal curve as a model for data that happens to pile up symmetrically, with the caution that not everything is a bell. The Central Limit Theorem is the reason the bell keeps reappearing: as the sample size grows, the sampling distribution of an average approaches a normal curve whatever the shape of the thing being averaged, with — in the standard statement — a "variance that equals the original variance divided by the sample size." Your p̂ is an average of 0/1 flips, so for a large enough n its sampling distribution is approximately normal, centred on p, with standard deviation SE.

That hands back the 68–95–99.7 rule, now pointed at the estimate: across re-runs, p̂ lands within one SE of p about 68% of the time, within two SE about 95%. The simulation agrees — 96% of the 200,000 re-runs fell within two SE of the true rate, near enough the 95% the rule promises. Which is the handle the next chapter grabs. If p̂ is within two SE of p about 95% of the time, then reaching two SE out from the p̂ you actually measured is a way to bracket the p you cannot see.

How wide is your uncertainty

The last chapter left p̂ scattering around the true rate in a bell, two standard errors of which cover about 95% of the runs. Turn that around and you get the interval every pass rate should be reported with.

From "95% of runs" to "95% confident"

If p̂ lands within two SE of p about 95% of the time, then the stretch from p̂ − 2·SE to p̂ + 2·SE reaches back over p about 95% of the time. That stretch is a confidence interval (CI). The textbook recipe is p̂ ± z·SE, where z is how many standard errors you want on each side — z ≈ 1.96 buys 95% — and this particular recipe is called the Wald interval.

Run it on 24/30. p̂ = 0.80, SE = 0.073, and 1.96 × 0.073 = 0.143:

text · the naïve textbook (Wald) interval on 24/30

Wald:  0.80 ± 1.96 × 0.073  =  [65.7%, 94.3%]

So "80% on 30 cases" is really "somewhere from about 66% to 94%, with 95% confidence." Before trusting that interval, two things need saying: what "95% confidence" actually claims, and why this particular recipe is the wrong one.

What the 95% is a statement about

The tempting reading is "there is a 95% probability the true rate is inside this interval." That reading is wrong, and it is worth being exact about why, because the error has the same shape as one "Shape and change" already warned about in "When one fact changes another" — swapping a conditional probability for its reverse.

The true rate p is a fixed number. It is not random; it does not have a 95% chance of being anywhere. The random thing is the interval, which jumps around every time you re-run the eval. "95% confidence" is a statement about the procedure: "were this procedure to be repeated on numerous samples, the proportion of calculated 95% confidence intervals that encompassed the true value of the population parameter would tend toward 95%." Ninety-five percent of the intervals the method produces contain p; you cannot know whether the one in front of you is one of them. The confidence is in the factory, not in the single part it stamped out.

Why the textbook interval is the wrong one

The Wald recipe has two flaws that both bite at the sample sizes evals actually use. It estimates the SE from p̂ itself, which is off when p̂ is near the edge; and it adds that error symmetrically, assuming the bell is even, when a rate near 0 or 1 has a skewed sampling distribution because there is less room on the side toward the boundary. So Wald's real coverage drops below the 95% it advertises, and near the top it produces intervals that leave the range of possibility. An agent that passed 28 of 30:

text · Wald degenerating near the ceiling

Wald on 28/30:  [84.4%, 102.3%]

A pass rate of 102.3% is not a wide interval; it is a broken one.

The fix is the Wilson score interval, which instead of estimating the SE and adding it symmetrically asks the inverse question — which true rates p would make an observed p̂ this far off plausible — and solves for the answer directly:

text · the Wilson score interval

             p̂ + z²/(2n)  ±  z·√( p̂(1−p̂)/n + z²/(4n²) )
Wilson  =  ─────────────────────────────────────────────
                          1 + z²/n

It is asymmetric, it never leaves [0, 1], and its coverage holds at small n. Wilson on the same 24/30 gives [62.7%, 90.5%] where Wald gave [65.7%, 94.3%] — pulled down and off the ceiling, which is the honest direction.

This is not a stylistic preference. Brown, Cai and DasGupta, reviewing interval methods for a proportion, found the Wald interval's "chaotic coverage properties … far more persistent than is appreciated" and recommend Wilson. And for this exact use, Bowyer, Aitchison and Ivanova's Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints finds the textbook methods "perform very poorly, usually dramatically underestimating uncertainty (i.e. producing error bars that are too small)," with "only the Bayesian credible interval and Wilson confidence intervals" holding their coverage at small N.

The width you are actually working with

Here is the Wilson interval on an observed 80% as the set grows from ten cases to five hundred:

text · 95% Wilson interval on an observed 80% pass rate, by sample size

    n =  10   observed   8/10  = 80.0%   95% CI [ 49.0%,  94.3%]   width 45.3
    n =  20   observed  16/20  = 80.0%   95% CI [ 58.4%,  91.9%]   width 33.5
    n =  30   observed  24/30  = 80.0%   95% CI [ 62.7%,  90.5%]   width 27.8
    n =  50   observed  40/50  = 80.0%   95% CI [ 67.0%,  88.8%]   width 21.8
    n = 100   observed  80/100 = 80.0%   95% CI [ 71.1%,  86.7%]   width 15.5
    n = 500   observed 400/500 = 80.0%   95% CI [ 76.3%,  83.3%]   width  7.0

Read the top row and the reflex to celebrate a two-point improvement should die on the spot. A 30-case eval reporting "80%" is reporting "somewhere between 63% and 90%" — a spread 28 points wide. It takes 500 cases to pull the interval in to plus-or-minus three-and-a-half points.

The width column is the 1/√n law from the last chapter seen from the side: it roughly halves each time n quadruples — 45 points at n = 10, about 22 at n = 50, 7 at n = 500. (The halving is only approximate near the top, because Wilson is deliberately lopsided up there.)

One caution, because it is the rule you were probably taught. General statistics says a sample above about 30 makes the normal approximation safe. For a binary outcome sitting up at 80–95%, that rule is exactly wrong: coverage stays poor well past 30, which is the whole reason the eval literature reaches for Wilson. "n > 30" is a fact about averaging arbitrary data, not about counting successes near a boundary.

Two numbers and a difference

You change the prompt, re-run the set, and the pass rate goes from 24/30 to 26/30. That looks like progress. After the last two chapters you can say why it is not yet a measurement, and this chapter is about the second thing those chapters set up: not how wide one number is, but whether the gap between two of them is real.

The gap has its own interval

Two overlapping intervals do not tell you the difference is zero, and two that clear each other do not tell you it is real. The overlap of two intervals is not the interval of their difference — you compute that one directly. Wilson's small-sample partner for the difference of two rates is Newcombe's method:

text · the interval on the change, not on either rate

  baseline  24/30 = 80.0%   Wilson 95% CI [62.7%, 90.5%]
  new       26/30 = 86.7%   Wilson 95% CI [70.3%, 94.7%]
  difference     = +6.7%    Newcombe 95% CI [−12.8%, +25.7%]   width 38.5 points

The honest reading of "24/30 went to 26/30" is: somewhere between a thirteen-point regression and a twenty-six-point improvement. The interval on the difference straddles zero, so the run is equally consistent with the change having done nothing, or having made things worse.

The question underneath is a hypothesis test

Behind that interval is a question with a standard shape, and it is worth naming because the rest of the chapter is answers to it: could a gap this big be nothing but sampling noise?

The way statistics asks it is deliberately backwards. Assume the boring thing — that the change did nothing, that the two prompts have the same true rate. That assumption is the null hypothesis. Now compute the probability of seeing a gap at least as big as the one you got, if the null were true. That probability is the p-value. A small p-value means your result would be a surprise under "nothing changed," which is evidence something did; a large one means what you saw is an ordinary night's noise. It is the complement-rule habit from "Chance: counting the ways" pointed at a claim: assume nothing happened, and measure how strange the data looks from there.

Running the same cases against both prompts sharpens the question, because you can compare case by case and throw away every case both versions got right; those carry no information about the change. Only the discordant cases — one prompt passed, the other failed — say anything. Under "no change," each discordant case is a coin flip about which prompt wins it, so the whole test collapses to a coin-fairness check on the discordant pile. That is McNemar's test, and it is Miller's recommendation for evals: "conducting statistical inference on the question-level paired differences, rather than the population-level summary statistics."

Pairing helps. It does not manufacture signal that is not there. A two-case net gain on 30 paired cases, in the most favourable split imaginable — nothing regressed, two cases newly passed — gives an exact two-sided p of 0.50:

text · McNemar exact test, the best case a 2-case gain can be

discordant pairs: 2 (both flipped the new prompt's way, none against)
McNemar exact two-sided p = 0.50

A p-value of 0.50 is a coin toss. The most favourable two-case improvement 30 cases can show you is, statistically, no evidence at all.

What it would take

So how many cases would it take? That depends on how big a change you are trying to catch, and the two levers are the size of the effect and whether you paired:

text · cases needed, two-sided, α = 0.05, power = 0.80

INDEPENDENT cases per arm            PAIRED cases (McNemar), total
  70% → 80%:   294                     20% discordant, 3:1 odds:  145
  70% → 75%:  1251                     20% discordant, 4:1 odds:   97
  80% → 90%:   199                     30% discordant, 3:1 odds:   97
  85% → 90%:   686                     30% discordant, 4:1 odds:   65

Three pieces of notation, once. Power = 0.80 means that if the change is real, this many cases give you a four-in-five chance of detecting it. Discordant pairs are the cases the two prompts disagreed on. 3:1 odds means three of those flipped your way for every one that flipped back. Detecting a ten-point change needs roughly 294 cases unpaired, or 97 paired.

Miller's worked example lands in the same place from the other direction: "new evals should contain at least 1,000 questions" to resolve a 3% difference. Both kinds of number are normal approximations, so read them as lower bounds — the true small-sample requirement is larger.

Practically: with 30 cases you can detect a prompt change that flips a third of your cases and wins three-to-one. Nothing subtler. That is the size of change a broken prompt produces, not the size of a tweak. An eval set is an instrument, and a 30-case set has a resolution you can now state in a sentence.

Spend the budget on cases, not trials

The instinct once you accept the noise is to beat it down by re-running: same cases, five trials each, average the result. It works on part of the noise and does nothing to the rest, and one formula shows exactly which part.

text · Miller's sample-size formula — watch where K appears

    n = (z_{α/2} + z_β)² · (ω² + σ²_A/K_A + σ²_B/K_B) / δ²

K is the number of trials per question. It divides σ², the within-question variance — the run-to-run wobble on one fixed case. It never touches ω², the between-question variance — the fact that some cases are harder than others. No number of trials shrinks ω²; only more questions do. Re-weighing the same parcel ten times gives you its weight precisely and tells you nothing about the average weight of parcels.

Over a fixed budget of 30 runs, split four ways, the standard error of the mean pass rate splits with it:

text · SE of the mean pass rate, 30 runs, p = 0.8 (variance 0.16)

    30 cases × 1 trial :   within-case SE 0.0730    between-case SE 0.0730
    15 cases × 2 trials:   within-case SE 0.0730    between-case SE 0.1033
    10 cases × 3 trials:   within-case SE 0.0730    between-case SE 0.1265
     6 cases × 5 trials:   within-case SE 0.0730    between-case SE 0.1633

If every case were equally hard, the split would not matter: the within-case column is flat at 0.073, because total trials stay at 30. Cases are never equally hard, and the between-case column is what you actually pay — it only falls as you add cases. Trade six cases run five times each for thirty cases run once, and you have more than doubled the standard error of the number you report.

Trials still earn their keep at a different job — finding which specific cases are unstable, a debugging signal rather than an aggregate one. As METR put it, "very high variance between runs on the same task may indicate a problem." That is a per-case flag to go read a trace, not a reason to re-run the whole set.

pass@k and pass^k are opposite tails

When you do run trials, pick the statistic that matches what you need, because two of them point in opposite directions. pass@k is "at least one of k attempts succeeded" — best-of-k, lenient, the right metric when a human will retry a bad answer. pass^k, from τ-bench, is "the chance that all k i.i.d. task trials are successful" — all-of-k, a reliability metric, the right one when the agent writes rows into a user's account unsupervised and every step has to hold. The arithmetic of the second is unkind. It is the multiplication rule from "Chance: counting the ways" turned on the agent: independent steps multiply, so on the plug-in estimate p^k, a 90%-per-step agent is 0.9^5 ≈ 0.59 over five steps. Ninety percent per step is a 59% agent end to end.

Why nobody gates a commit on this

The reason a full eval does not run on every commit is not flakiness. It is money, and the arithmetic is worth doing once with today's prices.

Take a 30-case run against Opus 5 (claude-opus-5, priced at $5 per million input tokens and $25 per million output). From the instrumented run, one case drives three model requests at roughly 5,400 input and 600 output tokens all told. So the cost of one case is:

text · per-case cost, Opus 5 at $5 / $25 per million tokens

  5,400 input  × $5  / 1e6  = $0.027
    600 output × $25 / 1e6  = $0.015
  per case                  = $0.042

That $0.042 is the per-case figure. The whole 30-case run is 30 × $0.042 = $1.26, an aggregate over the run and not a per-case cost — a distinction worth keeping straight, because the next paragraphs multiply it. Run the suite on every commit at twenty commits a day and the aggregate climbs to $25.20 a day, $756 a month.

That, not flakiness, is why eval suites do not gate commits. What teams do instead: a fast deterministic subset on every commit with no model calls at all, the full run nightly or on a release branch, and a gate on catastrophic regression rather than any regression. Which is the floor from "Assertions before judges," now with a reason attached.

How big should the set be, then?

Anthropic contradicts itself here, which is the clearest sign the question is unsettled. Their docs say "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals," with worked examples of 100 to 1,000 cases. Their engineering blog, in January 2026, says the opposite, and resolves it by naming the variable everyone leaves out:

We see teams delay building evals because they think they need hundreds of tasks. In reality, 20-50 simple tasks drawn from real failures is a great start. After all, in early agent development, each change to the system often has a clear, noticeable impact, and this large effect size means small sample sizes suffice. More mature agents may need larger, more difficult evals to detect smaller effects, but it's best to take the 80/20 approach in the beginning.

The number was never the question. Early on, changes are large and 20–50 hand-read cases resolve them; later, changes are small and you need hundreds, exactly as the sample-size table said. Start at thirty you personally looked at, and write down what that set cannot see.

Judging with a model

What survives the deterministic evaluators is the residue that is genuinely about language: whether the reply named the goal it attached the task to, invented a commitment the user never made, or asked a follow-up question when it already had everything it needed. You cannot enumerate the acceptable phrasings, so you hand the judgement to a model. pydantic_evals ships one, and reading what it sends is the fastest way to learn to write your own — captured verbatim from a real run:

text · the built-in judge's system prompt, printed from the library

You are grading output according to a user-specified rubric. If the statement in the rubric is
true for the provided input, expected output, and output, then the output passes the test. You
respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}

Examples:

<Input>What color is the sky?</Input>
<Output>Cerulean</Output>
<ExpectedOutput>Blue</ExpectedOutput>
<Rubric>The output is consistent with the expected output but doesn't have to match exactly</Rubric>
{"reason": "'Cerulean' is a shade of blue", "pass": true, "score": 1.0}

<Input>How many legs does a spider have?</Input>
<Output>Six</Output>
<ExpectedOutput>8</ExpectedOutput>
<Rubric>The output is factually consistent with the expected output</Rubric>
{"reason": "Spiders have 8 legs", "pass": false, "score": 0.0}

The "reason" field must be a concise 1-2 sentence justification. Do not include your reasoning
process, self-corrections, or re-checking in the reason. State only the final justification.

Four decisions are visible there, and they are most of "how to write a judge that is not garbage," made for you already.

reason comes first in the schema, and field order is generation order for a structured output — the library's GradingOutput really does emit reason, then pass, then score. So the judge writes its justification before it commits to a verdict: chain-of-thought by construction, not by instruction.

The reason is constrained to one or two sentences, because a rambling judge produces reasons nobody reads, and the reason is the only part you can audit.

The rubric is a proposition, not a question, so write yours as an assertion the judge can only pass or fail: "The reply states, in one sentence, exactly which task was created and which goal it was attached to. It does NOT ask a follow-up question."

And there are two few-shots, one pass and one fail, so the judge carries no prior about which verdict you expect.

One nuance the library hides, and it matters if you copy the prompt above. That four-section prompt, with its Input, Output, ExpectedOutput and Rubric sections, is only one of several the library keeps, and you get it only by asking for both the input and the expected output: LLMJudge(include_input=True, include_expected_output=True). The defaults are both False, which dispatches to a leaner prompt that shows the judge only the output and the rubric; ask for the expected output alone and you get a variant with no <Input> block at all. Know which one you are actually sending. One more version fact worth stating: the library's default judge model is openai:gpt-5.2, an OpenAI model and not a Claude one, so override it deliberately.

Binary, not one-to-ten

The default output is an assertion with a written reason and no number, and that default is right. Measurements of judge self-consistency have found models far steadier on a binary or relational judgement than on a 1–5 scale, drifting toward the middle of the scale when unsure — one reason a Likert score is a worse instrument than it looks. Husain puts the prescription bluntly: "If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong," because "people don't know what to do with a 3 or 4." Agent tasks do have partial credit — express it as several binary assertions, not one graded score.

The biggest lever is a reference answer

It is not the rubric. Doddapaneni and colleagues perturbed 2,400 answers and measured how often judges missed the quality drop: a plain single-answer prompt failed 57% of the time, an elaborated rubric 85% — worse — and a prompt shown a reference answer only 26%. MT-Bench found the same independently: on maths problems the judge could solve but got wrong because it was "misled by the provided answers," reference-guided grading cut the failure rate from 70% to 15%.

It is one keyword — LLMJudge(include_expected_output=True) — and every case promoted from a real failure already has an expected_output, because a human wrote it during triage.

The named biases, with their measured sizes

These are all from MT-Bench, on 2023-era models, so treat the directions as established and the magnitudes as dated:

Bias	What it is	Measured
Position	"a propensity to favor certain positions over others"	swap the two answers and GPT-4 repeats its verdict only 65.0% of the time — the best of three judges tested
Verbosity	favouring "longer, verbose responses, even if they are not as clear … or accurate"	under a padding attack, Claude-v1 and GPT-3.5 preferred the padded answer 91.3% of the time; GPT-4, 8.7%
Self-enhancement	"LLM judges may favor the answers generated by themselves"	GPT-4 "favors itself with a 10% higher win rate"; Claude-v1, 25%

The 65% is the one to sit with. A judge that changes its mind a third of the time when you reorder the inputs is not measuring the inputs. The paper is fair — "this test is challenging because the answers are very similar and occasionally indistinguishable even to humans" — and the remedy is mechanical: run each comparison twice with the order swapped and count only the consistent verdicts. The order-of-magnitude spread between judges on the same verbosity attack, 91.3% against 8.7%, is itself the argument for testing yours instead of assuming it.

Judging the judge

Here is the half that gets skipped. You now have a second model, with its own failure modes, grading against a rubric you wrote badly the first time. Before its numbers mean anything it needs a hand-labelled holdout, a measured agreement rate, and — first — a correct idea of what agreement is even achievable.

The ceiling is not 100%

Everybody quotes MT-Bench's ">80% GPT-4 agreement with humans" and drops the sentence beside it: "The agreement under setup S2 (w/o tie) between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%)."

Humans agreed with each other 81% of the time. Without that comparison the 85% is marketing.

And that baseline is a property of the task, not a constant. On a different task, Thakur and colleagues measured human–human agreement at 98.52% ± 0.42% — Scott's π of 96.2 — with their best judge around 88 π, "8 points behind human judgment." On an easy task a judge at 88 is poor; on a hard one a judge at 80 is at parity. Measure the baseline on your own task before you read your judge's score.

The κ numbers in circulation are wrong

Thakur et al. is widely cited for Cohen's κ figures — GPT-4 at 0.84, human–human at 0.97. The authors withdrew them: "In an earlier version of this paper, we used Cohen's kappa … It has since come to our attention that – despite it's widespread use – this metric has some well-documented theoretical issues."

κ "adjusts for the observed average differences between raters, which is in fact part of what we intend to measure"; Scott's π "defines the chance baseline under the assumption that the raters have the same distribution." Use π, cite v6, and distrust write-ups carrying the old numbers. Watch the scale when you do: κ is reported on 0–1, π on 0–100, and they measure the same axis.

Raw agreement is mostly a measure of your class balance

The line to keep, from the same paper: "while Llama-3 8B has a Scott's π score of only 59, its percent agreement is still well above 80%." A judge agreeing with you four times in five sounds usable. But where most answers pass, a judge that always says "pass" scores about that too — it has learned your class balance, not your rubric.

Hence Husain's advice not to report agreement at all: "Using raw agreement is generally not recommended … you should typically measure precision and recall separately." pydantic_evals ships exactly the tools for it as report evaluators — PrecisionRecallEvaluator, ConfusionMatrixEvaluator and ROCAUCEvaluator, which run over the whole labelled experiment rather than one case. Their positive_from='labels' parameter is the hook: score the judge's verdicts against your hand labels, and read precision and recall instead of a single agreement number.

You can overfit a rubric

Iterating a judge prompt against thirty examples until it agrees with you is fitting a model to thirty examples, with everything that implies. Eugene Yan built the dev/test split into a tool and watched it happen: at 25 items per split, "the LLM-evaluator may overfit on the dev set and fail to generalize to the test set." Hold labels out, and aim for 50–100 before believing anything the judge tells you.

One worry you can drop: which model judges is not a cost question. The judge reads one case's input and output and writes a sentence — far less work than the agent's three requests. At Haiku 4.5's prices (claude-haiku-4-5, $1 / $5 per million tokens), one judge call per case adds a few cents across the whole 30-case run, on the order of 4% on top of that run's $1.26; an Opus 5 judge, priced like the agent itself, adds closer to a fifth. Either way the delta is cents on the run. Choose the judge on measured agreement and bias, and ignore the price.

/ closing the loop

Feedback into eval cases

Thirty cases you invented measure your imagination. The set only starts measuring your users when the users put things in it — the third of the three moves, and the reason the first two came first.

Not all signals are worth the same

The asymmetry to internalise: negative signals are informative and positive ones are not. A thumbs-down happened because something specific went wrong; a thumbs-up happened because nothing did, which you had already assumed.

Signal	Worth	Why
The user edited what the agent created	Highest	A correction, not an opinion: the diff between what the agent wrote and what the user wanted is the label, and it costs the user nothing extra.
The user deleted it within minutes	High	Unambiguous rejection with a timestamp. Deletion weeks later is just life.
The user re-asked, rephrasing	High	A retry is a failure the user was too polite to report.
It still exists, or got completed, a week later	High but slow	The only signal measuring usefulness instead of plausibility.
Explicit thumbs down	Medium	Rare, sparse, and sometimes means "I changed my mind." Its value is as a pointer to a trace worth reading.
Explicit thumbs up	Low	Given by people already happy, absent from the silent majority of good turns.
Session length, turn count	Low	Confounded both ways: a long session is engagement or floundering.

That ranking is an argument from signal-to-noise, not a published finding — a starting hypothesis to check against your own data.

What has to be stored for a label to survive six weeks

text · the columns; the last four are filled in by a human, later

id                  uuid
created_at          timestamptz
user_id             fk           -- who gave it
turn_id             fk           -- YOUR id for the turn, minted before the model call
trace_id            text         -- 32 hex, W3C; joins to the observability backend
span_id             text         -- 16 hex; the invoke_agent span specifically
prompt_version      text         -- your own prompt-version span attribute; no standard gen_ai one
app_version         text         -- commit SHA
signal_kind         enum         -- thumbs | edit | delete | reask | implicit_kept
signal_value        smallint     -- -1 / +1, or null for implicit
target_kind         enum         -- task | goal | journal_entry | turn
target_id           uuid         -- the row the agent created, if any
user_comment        text         -- nullable; PRIVATE, same class as journal content
reviewed_at         timestamptz  -- nullable; when a human triaged it
review_verdict      enum         -- agent_wrong | agent_right | ambiguous | user_error
expected_behaviour  text         -- the human's answer. THIS is what becomes a Case.
promoted_case_id    text         -- nullable; the Case name once promoted

Three are the ones people get wrong. Mint turn_id yourself, before the model call — do not make the UI wait on a trace id to have something to attach a thumbs to, and do not assume the tracing backend still holds the trace when you come to read it.

A prompt version is not optional. A label with no prompt version is a label you cannot use, because you cannot tell whether it describes behaviour you have already fixed. There is no standard gen_ai.* attribute for it, so this is a column you own and fill yourself.

And user_comment is user content, on the journal's side of the privacy line.

The join key is the W3C trace id, and the plumbing is three lines run inside the turn: trace.get_current_span().get_span_context() gives a 32-hex trace_id and 16-hex span_id to store on your own row. Pydantic AI also propagates a run id as OTel baggage, so a downstream service can log it without being handed it — but baggage is scoped to the run and empty afterwards, which is another reason the ids live on your row.

Promotion is four fields

Cases are data, and pydantic_evals treats them as such: Dataset.to_file() writes YAML with a sidecar JSON Schema and a yaml-language-server directive so your editor validates it.

yaml · a promoted case, round-tripped through to_file/from_file

# yaml-language-server: $schema=planner_schema.json
name: planner_from_production
cases:
- name: thumbs_down_2026_08_19_a41f
  inputs: move my gym sessions to mornings
  metadata:
    source: user_feedback
    trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
    span_id: 00f067aa0ba902b7
    label: thumbs_down
    labelled_by: reuben
    labelled_at: '2026-08-19T14:02:11Z'
    note: created three duplicate tasks instead of editing
  expected_output: updates the existing tasks rather than creating new ones
  evaluators:
  - Contains:
      value: updated
      as_strings: true
evaluators:
- IsInstance: str
- MentionsGoal: gym

inputs is the original user message, metadata carries the trace id back to the turn, and expected_output is the one field a human must supply — what should have happened. No tooling removes that one.

Hence the corollary: the app writes candidates into a queue a human drains; it must never write eval cases directly. Make the queue fast and keyboard-driven, and make it so the criteria can grow — criteria drift says they will.

What a few hundred labels actually buys

They find your failure modes, which takes about twenty labels, not hundreds — twenty bad turns tell you the three things really wrong. They populate an eval set: 200 labels triaged to perhaps 60 promotable cases. They supply few-shot examples, at the price of prompt tokens on every request forever, and past five or ten they crowd more than they help.

They will not fine-tune anything — not the volume, and more importantly not the shape: these are preference labels on a trajectory, not supervised pairs. They will not give you a product-quality metric either, because a few hundred self-selected labels are not a sample of traffic; they are a sample of turns that annoyed someone enough to click.

Your pass rate over promoted cases is a pass rate over your hardest cases by construction. It will look terrible, and it should — and after the statistics chapters you can now report it with an interval, watch that interval tighten only as the set grows, and gate the release on a floor rather than on the last two cases that flipped.