An Eval Inherits Every Blind Spot in Its Dataset and Its Grader

A measurement loop looks like rigor. You run the current version of a prompt against a set of cases, score each output, and compare the aggregate to the last version. The number that comes out feels objective in a way a reviewer’s impression never could. That feeling is mostly borrowed. An evaluation does not manufacture its own trustworthiness. It inherits it from two ingredients that sit underneath the loop and rarely get the same scrutiny the loop itself does. One is the dataset the prompt runs against. The other is the grader that turns each output into a pass or a fail. A flawless loop wrapped around a thin dataset or a biased grader produces confident, precise, well-documented numbers that happen to be wrong. Both ingredients are systems in their own right, with their own failure modes, and both deserve the engineering discipline usually reserved for the thing they measure.

The dataset draws the boundary of what an eval can see

An eval can only make a statement about the cases it contains. Everything outside the dataset is invisible to it, and the number it produces says nothing about that unseen space no matter how authoritative it looks. This is the property that makes dataset construction the upper bound on everything the workflow can conclude: the measurement is exactly as good as the set of questions it asks, and not one case better.

Coverage is the first axis, and it has three distinct bands a dataset has to reach on purpose. Typical inputs are the ones the system meets under normal conditions, the cases a prompt was obviously built to handle. They are necessary and they are the least informative, because a prompt that fails the typical case fails on the first day and never ships. Edge cases sit at the boundary of the intended scope: inputs that are odd, underspecified, or malformed, sitting right where the prompt’s intended scope runs out. Adversarial inputs go further, constructed deliberately to break the thing, to trigger a refusal where none belongs, to force malformed output, to find the handling that quietly falls apart. The failures that do the most damage in production almost never live among the typical cases. They live in the two bands a casual reviewer would never think to write down, which is why a dataset drawn only from typical inputs delivers the same untested optimism a handful of manual glances would, except it now arrives formatted as a rate and is that much harder to second-guess.

Volume is the second axis, and it is easy to underweight because a small dataset still returns a clean-looking number. It just returns one that cannot bear any weight. When a set is small enough that a single flipped case moves the headline rate by several points, the rate is not measuring the prompt so much as sampling noise, and a deployment decision resting on it is a decision resting on almost nothing. Statistical basis is not a nicety here. It is the difference between a number that discriminates between two prompt versions and a number that merely looks like it does.

Three sources, and none of them is sufficient alone

Cases come from three places, and the reason mature suites draw from all three is that each source is strong exactly where the others are weak.

Handcrafted cases are written by someone who understands what the prompt is supposed to do and where it is known to break. They are the most precise instrument available, because a human author can target a specific failure mode on purpose. They are also the slowest to produce, and human authors share blind spots with the prompt they are testing. You tend not to write the case you did not think of, which is the same case the prompt was not built to handle.

Synthetic cases are generated by a model from a seed of topics, patterns, or example inputs. Their advantage is volume and surface area. A generator can produce breadth a human would never sit down to enumerate, and it can vary phrasing and framing across a space far wider than handcrafting reaches. The cost is quality control. Generated cases skew toward the redundant and the plausible-but-shallow, and a synthetic set accepted without review inflates the size of the dataset without proportionally expanding what it actually tests. Synthetic generation extends coverage only when a human filters what it produces.

Production-sampled cases are real inputs pulled from the running system, labeled after the fact. They carry a validity the other two cannot, because they are not a guess about what users will send. They are what users actually sent. Their limitation is chronological. You cannot sample production before you are in production, so they arrive last, and they arrive shaped by the traffic you already have rather than the traffic you are about to attract. The standard path reflects that ordering. Bootstrap with a handcrafted set to get a precise starting instrument, widen it with synthetic generation to buy coverage, and fold in production samples once the system is live and telling you where reality diverges from what you imagined.

The deeper point underneath the three sources is that a dataset is never finished. It has to keep pace with an input space that shifts as the product, its users, and the conditions around them change. Freeze it and a slow corruption sets in: repeated tuning against the same unchanging cases bends the prompt toward those cases in particular, so the score climbs while the behavior it is meant to stand in for drifts out from under it. Each failure that first surfaces in production is evidence of a case the set never held, and returning that case to the set is what keeps the eval pointed at reality rather than at its own history.

Two ways to turn an output into a verdict

The dataset supplies the questions. The grader supplies the answers, and there are two fundamentally different machines for producing them, separated by whether correctness can be written as a rule.

Code-based grading applies wherever correctness is programmatically definable. It is fast, deterministic, and cheap enough to run at scale without a second thought, and where it applies it is always the right choice for exactly those reasons. It comes in a few shapes of increasing flexibility. Exact match checks an output against a fixed expected value and is perfect when the answer is deterministic and useless the moment acceptable outputs vary in wording. A pattern check widens that to any output describable by a rule, accommodating variation in form while staying fully deterministic. Schema validation asks whether an output is a valid instance of a defined structure, which is the natural grader for anything that is supposed to be machine-readable. Assertion-based checks are the most general: pass the output into a set of programmatic tests that verify whatever properties actually matter for the task. The through-line is that all of these are decidable in code, which is what buys the determinism.

The boundary of code-based grading is sharp. It only reaches output types where correctness reduces to a check a program can run. The moment the thing you care about is a judgment, whether a summary is faithful, whether a tone is right, whether an explanation is sound, there is no assertion to write, and the deterministic machine has nothing to grade. That is the space model-based grading exists to cover, and it covers it by handing the judgment to a separate model. Give a grader the original input, the output under evaluation, and a rubric describing what a passing response looks like, and let it return a verdict and a rationale. Everything rests on the rubric. Unless it pins down what a passing response looks like in terms concrete enough to mean the same thing on every case, the grader has room to improvise, and a grader left to improvise is not lenient, it is erratic.

A model grader is a system with its own reliability question

Introducing a model into the grading step solves the coverage problem and imports a new one. The grader is no longer a deterministic function. It is another nondeterministic system whose output you are now trusting, and it fails in ways code never does.

The first failure mode is bias toward its own defaults. A model grader tends to reward outputs that resemble what it would have produced, which means stylistic alignment can inflate a score even when the underlying quality does not justify it. The measurement acquires a preference of its own, and that preference contaminates every comparison run through it.

The second is the untested grading prompt, and it is the most easily overlooked because the grading prompt does not feel like part of the system under test. It is. A grading prompt is a prompt, subject to the same drift as any other, and if it changes without being evaluated the grader’s behavior shifts underneath a metric that is supposed to be holding still. Historical comparisons silently stop being comparable, because the ruler changed between measurements and nobody wrote it down. A grader whose prompt is not version-controlled and evaluated is a moving reference standard masquerading as a fixed one.

The third and most consequential is treating a model-graded score as ground truth. A number returned by a grader looks exactly like a number returned by a code check, and it is tempting to grant it the same authority. It has not earned it. Absent any anchor to human judgment, a model-graded score overstates its own precision. It is a confident estimate of quality with no established relationship to the quality a person would actually perceive. None of these three risks disqualifies model-based grading. Each of them says the grader has to be managed with the same discipline applied to the prompt it is judging, because it is the same kind of object.

Calibration is how a grader’s numbers earn their authority

The bridge between a model grader’s scores and something you can defend is calibration, and it is a concrete procedure rather than a posture. Sample a representative subset of the eval cases. Have humans label each one independently, without seeing the grader’s verdict, so the human judgment is a clean reference. Run the same cases through the grader. Then measure how often the two land on the same call. That agreement fraction is the number that tells you how far the grader’s scores can be trusted.

Agreement is not the end of the procedure. The disagreements are where the value is. Systematic divergence between grader and humans is diagnostic. It points either at a rubric that is underspecified, leaving room for the grader to apply a standard the humans do not, or at a grader reading the rubric differently than intended. Either way the remedy is to tighten the rubric to close the gap and run calibration again, until the grader’s verdicts track human judgment closely enough that you can state the relationship as a number rather than assume it. A calibrated grader is reliable at scale precisely because that relationship is known: its scores are no longer a hopeful proxy but a measured one. An uncalibrated grader does the opposite. It injects a silent, unquantified error into every downstream metric, and because the error is silent, every comparison built on top of it inherits a distortion nobody can see or correct for.

Confidence is manufactured upstream of the number

The measurement loop gets the attention because it is where the number appears, but the number’s honesty is decided before the loop ever runs. A dataset that fails to reach the edge and adversarial bands measures a prompt only on the inputs least likely to break it. A grader that has never been calibrated reports a precision it does not possess. In both cases the loop faithfully computes and compares, and faithfully produces a result you should not act on, because the flaw is not in the arithmetic but in what the arithmetic was fed.

There is no version of this that escapes human judgment. Handcrafting a case, deciding a synthetic set is representative, writing a rubric, labeling a calibration sample: each is a point where a person’s understanding enters the system, and the value of the workflow is not that it removes those judgments but that it concentrates them where they can be examined and then amortizes them across every future run. A dataset built with care and a grader calibrated against people are what let the eval loop mean what it appears to mean. Skip that upstream work and the loop still runs, still produces its confident percentage, and tells you nothing you should trust. The rigor of a measurement is never in the measuring. It is in what you chose to measure and who decided what counts as right.