Your eval set is too small, and it is costing you launches

Almost every team we meet has an eval set. It usually lives in a spreadsheet, it usually has between 20 and 60 rows, and it was usually written in an afternoon by whoever was closest to the problem. It is genuinely better than nothing. It is also too small to answer the question you are actually asking it.

The question is not “did this prompt change help?” The question is “is this prompt change safe to ship to everyone?” Those need very different amounts of evidence.

Forty cases cannot see a five-point move

Suppose your agent passes 80% of a 40-case set. You change the prompt and it now passes 85%. Two more cases went green. That feels like progress, and someone will want to ship it.

But the 95% confidence interval on 80% of 40 samples runs roughly from 65% to 90%. The interval on your new 85% overlaps almost entirely. You have not measured an improvement; you have measured the same system twice and got two numbers. To resolve a five-point difference with any confidence you need several hundred cases, not forty.

A useful rule of thumb: if you cannot state the confidence interval on your eval score, you cannot tell an improvement from noise — and you will eventually ship a regression because the dice came up green.

Small sets fail in a second, worse way

Sample size is the obvious problem. Coverage is the expensive one. Hand-written cases cluster around what the author could imagine at the time, which is almost always the clean, central, well-formed version of the task. Production traffic does not look like that.

• The inputs that break things are long, contradictory, or half-empty — the customer who pastes an entire email thread, the form submitted with three fields blank.
• The expensive failures are concentrated in segments that are small in volume: one region, one document type, one enterprise customer with an unusual schema.
• Nobody hand-writes the adversarial case, so prompt injection and jailbreak coverage is typically zero until an incident creates it.

An eval set that is 90% happy path will report a healthy score while the system fails for the users who complain loudest.

Where the extra cases come from

You do not write your way to a thousand cases. You harvest them.

1. 01Log everything from day one — inputs, retrieved context, tool calls, the final output, and any downstream signal such as an edit, a retry, a thumbs-down, or an escalation to a human.
2. 02Sample stratified, not uniformly. Bucket traffic by the dimensions that matter to your business — customer tier, document type, language, length — and pull cases from every bucket, including the thin ones.
3. 03Mine the failures you already have. Support tickets, escalations and manual corrections are pre-labelled examples of your system being wrong, and they cost nothing to collect.
4. 04Grow the set on every incident. Any bug that reaches production earns a permanent regression case. This is the single habit that separates teams whose quality compounds from teams who keep re-fixing the same class of failure.

Grade the thing you actually care about

A single pass/fail number hides the tradeoff you are usually making. Split the score into the dimensions that can move independently — factual correctness, format compliance, tone, refusal behaviour, latency and cost — and track them separately. Most “improvements” we see are a real gain on one dimension paid for by a quiet loss on another, and an aggregate score will not show you the trade.

Model-graded evaluation is fine for the fuzzy dimensions, provided you validate the grader itself against a few hundred human judgements and re-validate when you change grader models. An unvalidated LLM judge is a random number generator with good manners.

What good looks like

• Several hundred to a few thousand cases, drawn from real traffic and refreshed on a schedule.
• Explicit slices, each with its own score, so a regression in a small segment cannot hide inside the average.
• Every past incident represented as a permanent regression case.
• Evals in CI, gating deploys — not a notebook someone runs when they remember.
• Confidence intervals on the headline number, so “better” means better.

None of this is exotic. It is a week of engineering that most teams keep deferring because the spreadsheet is technically working. The cost of deferring is not that you ship something bad — it is that you stop being able to ship at all, because nobody can prove a change is safe, so every launch turns into an argument.