Reviewer Attention Is a Budget, and Random Sampling Spends It Badly

Every system that puts model output in front of a human reviewer is running a sampling policy, whether or not anyone designed one. Review capacity is finite and output volume is not, so some rule decides which fraction of the output a person actually looks at. The default rule, and the one that feels most defensible because it feels most neutral, is to draw that fraction at random. It is the wrong default. A uniform random sample reproduces the distribution of the output, and in any system healthy enough to be in production, the output is mostly correct. The sample inherits that property. Reviewers spend their scarce attention confirming what was already right, the failures arrive too sparsely to form a pattern, and the program can report a review rate while staying almost completely uninformed about how the system fails.

More sampling does not fix this. What fixes it is to stop treating the sample as a fair slice of the output and to start treating it as an allocation of a budget. Reviewer attention is the budget. The population of outputs is where it can be spent. The question a sampling policy answers is where a unit of attention buys the most, and the answer is almost never “spread it evenly.”

A random sample is a report on the output distribution

The arithmetic is unforgiving and it does not depend on any particular numbers. Whatever fraction of the output is wrong, that same fraction is what a uniform sample hands the reviewer, because a uniform sample is a scale model of the population. If the system errs one time in twenty, nineteen of every twenty items in the queue are confirmations. That ratio is fixed by the system’s accuracy, not by the sample size, so scaling the review program does not improve it. Review ten times as much and you get ten times the confirmations along with ten times the corrections, at ten times the cost. The error density of a random queue is exactly the system’s error rate, and there is no sampling rate that changes that.

The second failure is worse than the inefficiency, and it is the one that actually undermines the program. The few errors a uniform sample does turn up are scattered thinly across a population of tens of thousands, which leaves them as a set of unrelated incidents. They almost certainly come from different inputs, different segments, different causes. A reviewer can correct each of them, and the system is no wiser afterward, because a single instance of a failure mode is indistinguishable from noise. Diagnosis requires repetition. You learn that a model mishandles a class of input by seeing it mishandle that class several times, which lets you separate the pattern from the accident. Random sampling is structurally hostile to that. It scatters the errors it does surface across every cause at once, so the sample is rich in incidents and poor in evidence.

Underneath both problems is an allocation mistake that becomes obvious once it is named. Random sampling allocates review effort in proportion to volume. But the precision of what you learn about any segment of the population depends on how many samples you drew from that segment, not on how large that segment is. If you want comparable confidence about the model’s behavior in every segment, you need roughly comparable numbers of observations in every segment. Proportional allocation gives you the opposite: overwhelming precision about the segment that was already dominant and already understood, and essentially no resolution on the segments that are small, unusual, or new. Volume is a poor guide to where the remaining information sits, and proportional sampling is built to follow volume and nothing else.

One sample cannot serve both reasons to look

Before choosing a policy, it is worth separating two motives that get collapsed into the phrase “we review a sample of outputs,” because they want opposite designs and most review programs are quietly trying to serve both with one queue.

The first motive is assurance. You want to know how good the system currently is: a defensible number you can report, compare across releases, and hold against a commitment. Assurance wants an unbiased estimate of a population quantity, and for that purpose an unbiased sample is exactly right. Uniform sampling is the correct policy here, not a naive one.

The second motive is improvement. You want to find the failure modes, understand them, and fix them. Improvement does not want an unbiased picture. It wants the highest possible concentration of informative failures, clustered densely enough to be diagnosable. Deliberate bias toward the suspect regions of the population is not a flaw in that design. It is the design.

Trying to satisfy both with a single queue produces a policy that does neither well, and it usually fails in a specific direction: the queue gets tilted toward suspected failures for improvement’s sake, and then someone computes the system’s error rate from that same queue and reports a number that is badly pessimistic, because the sample was drawn from where the errors were expected to be. The reverse mistake is just as common. The queue is left uniform for the sake of a clean headline number, and the improvement program starves. The resolution is to run both, keep them separately labeled, and understand that they are answering different questions with different sampling logic. The improvement lane is where stratification earns its keep and where the design effort belongs, but the assurance lane has to exist somewhere, or you lose the ability to say how good the system actually is.

Stratification is an allocation policy over the population

Stratifying means partitioning the output population into groups that you expect to behave differently, then choosing a sampling rate for each group independently. That is the whole mechanism. Its power is entirely in the choice of partition, because the partition is where you encode what you believe about how the system fails.

One constraint governs which partitions are even available: the key you stratify on has to be computable at selection time, from the input and the output alone. You are choosing what to review before anyone knows whether it was right, so anything that requires ground truth to compute is disqualified. This sounds obvious and it eliminates the most tempting policies. You cannot sample the wrong answers preferentially, because identifying them is the thing you were going to pay a reviewer to do. Every usable stratification key is a proxy, a signal correlated with error or with the cost of error, available before the fact. The craft is in picking proxies that carry real signal, and in remembering that they are proxies.

Three families of key are worth building around, and they are worth separating carefully, because they are proxies for genuinely different things and teams routinely treat them as interchangeable.

Uncertainty is a proxy for the probability of error

The model’s own confidence is the most immediately available signal, and it is the natural first stratum. Outputs the model is unsure about are, in a system whose confidence means anything at all, the outputs most likely to be wrong. Sampling them at an elevated rate concentrates errors in the queue with no additional machinery beyond a threshold, and for most systems no other single change puts as many genuine errors in front of a reviewer for the same cost.

Two limits on it matter. The first is that confidence is only a usable stratification key to the degree it has been checked against outcomes. An unchecked score is the model’s opinion of itself, and a policy built on it inherits whatever distortion that opinion carries. Where the model runs systematically overconfident, its outputs get sorted into the low-priority stratum and reviewed least, so the policy looks away from exactly the region it should be staring at. Stratifying on confidence builds the model’s self-assessment into the process meant to audit the model, which turns calibration from an academic property into something the whole review design rests on.

The second limit is structural and it survives even perfect calibration. Confidence tells you where the model knows it is on thin ice. It is silent about where the model is wrong and does not know it, and confident errors are both the most damaging class of failure and the class a confidence-driven sampling policy will systematically fail to surface. A policy that draws its review population entirely from the low-confidence tail can only ever discover the failures the model already suspected. Everything it learns confirms that the high-confidence output is fine, because the high-confidence output is never examined. This is the sharpest reason a review program cannot be confidence-stratified alone.

Rarity is a statistical problem before it is a risk problem

The second family of key is the structure of the population itself: category, segment, tenant, input source, locale, document type, whatever natural partition your domain provides. Rare segments are the ones a proportional sample starves, and they deserve their own quota for a reason that is worth stating precisely, because it is often given as a risk argument when it is really a statistical one.

The argument usually offered is that rare segments have higher error rates, so they deserve more attention. Sometimes that is true, and when it is true, it is a good reason. But the more fundamental point holds regardless of whether the rare segment is actually worse: you do not know its error rate, and under proportional sampling you cannot find out. A segment producing a small fraction of your volume produces a correspondingly small number of reviews, and an error rate estimated from a handful of observations has an uncertainty band wide enough to be useless. That segment could be the best-performing part of the system or the worst, and your data cannot distinguish those cases. The model is unaudited there, and the review program’s own numbers will not tell you so.

That is what a per-segment quota buys. A quota fixes a minimum number of observations for each segment and refuses to let volume lower it, so resolution stays roughly even across the map instead of piling up in one corner of it. The floor is also what makes drift detectable at the segment level. A system’s quality does not usually degrade uniformly. It degrades in a corner, when an upstream input source changes format, a new tenant onboards with different conventions, or a category’s real-world distribution shifts underneath a model trained on the old one. Detecting that requires enough observations in the corner to distinguish a real change from noise, and proportional sampling guarantees you will not have them. Without a floor, a localized regression sits below the resolution of the review program entirely, and the first thing to surface it is whatever the failure eventually does downstream.

Consequence is the axis confidence cannot see

The third key is the one most often left out, and it is not a proxy for the probability of an error at all. It is the cost of one. The value of reviewing an output is the expected harm the review prevents, and expected harm is the probability that the output is wrong multiplied by what it costs when it is. Uncertainty and rarity are both attempts to estimate the first factor. Nothing about them says anything about the second, and the second varies across outputs by orders of magnitude more than the first ever will.

An output that commits an action nobody can walk back, that decides what a person is entitled to, that feeds a regulatory filing, or that propagates into a store many downstream consumers read, carries a cost of error that dwarfs a routine classification whose worst case is a minor annoyance corrected on the next pass. A modest probability of error on the former is worth far more reviewer attention than a high probability of error on the latter. The stakes axis therefore has to be able to override the confidence axis: an expensive failure earns review because of what it would cost, not because the model volunteered any doubt about it. That is not primarily a hedge against miscalibration, though it works as one. It follows directly from the fact that the confidence score only ever addressed one of the two terms in the product.

There is a second, compounding reason. Confidence is least trustworthy exactly where the input is unusual, because unusual inputs are the ones least represented in whatever shaped the model’s sense of its own reliability. High-stakes cases are frequently unusual cases: the edge condition, the exception, the escalated situation, the transaction that is large because something about it is atypical. The correlation runs the wrong way. The outputs whose errors cost the most are disproportionately the outputs whose confidence scores you should trust the least, which means the stakes axis is doing its most important work precisely where the uncertainty axis is quietly failing.

The three axes are terms in one estimate

With three keys, the temptation is to write three rules and let them fight over the queue. The cleaner formulation is to treat the sampling policy as a ranking function over the population, scoring each output by the expected value of reviewing it, which is the estimated probability that it is wrong multiplied by what being wrong would cost, adjusted by how much you still have to learn about the segment it came from.

Read that way, the three keys stop being competing rules and become the three terms of one estimate. Confidence supplies the probability. Stakes supply the cost. Segment coverage supplies the exploration term, the value of looking somewhere you cannot yet predict. An output that is uncertain, from a thin segment, and consequential scores highly on all three and belongs at the top of the queue. An output that is confident, from your dominant segment, and trivially reversible scores low on all three and can be sampled at a token rate. No universal weighting exists, because the exchange rate between the three terms depends on what the system does. The objective they serve does not vary at all: spend the budget where being wrong is most expensive, and where you know least about how often it happens.

In practice this resolves into a small number of lanes with explicit rates rather than a single continuous score, because lanes are auditable and a continuous ranking is not. A mandatory lane for high-stakes output, sampled at or near full coverage. A priority lane fed by low confidence and by segments below their quota floor. And a baseline lane, sampled uniformly, whose importance turns out to be wildly disproportionate to its size.

Concentration is a bet, and the bet has to stay falsifiable

Each of these mechanisms points attention at where you believe the failures are. That belief is a hypothesis, encoded in the strata and their rates, and it is the central weakness of the entire approach: a stratified policy can only ever find the failures it was designed to look for. Attention spent on the suspect regions is attention taken from everywhere else, so the outputs that are confident, common, and low-stakes get sampled at a lower rate than random sampling would have given them. If the real failure mode lives there, and confident errors in the dominant segment are exactly the failure mode most likely to live there, the stratified policy is worse than the naive one it replaced, and it will not tell you so.

Worse, the policy is self-confirming. The only data you collect about system quality comes from the strata you chose to sample, so the strata you deprioritized generate no evidence of failure, and their silence reads as health. That evidence then justifies keeping the rates where they are. A review program can run for a long time in this state, reporting steadily that the high-confidence output is fine, on the strength of never having looked at it.

The antidote is structural and cheap: keep a genuinely uniform random floor across the entire population, including the strata you believe are safe, and never let the optimization consume it. That floor is what makes the hypothesis falsifiable. It is the only mechanism that can surface a failure mode your stratification did not anticipate, and it is the sole source of an unbiased estimate of true system quality, which is the assurance lane from earlier, now doing double duty. Its rate can be low. What it cannot be is zero, and the pressure to zero it out is constant, because by construction it is the least productive lane in the program. That is the point. A lane that finds nothing is reporting a result, and the day it starts finding something is the day it justifies its entire existence.

Two further costs of stratification deserve naming plainly. Any quality number computed from a stratified sample is biased, and badly so, because the sample deliberately overrepresents the regions where errors concentrate. Reporting an error rate straight from a stratified queue overstates the failure rate of the system, sometimes by a large multiple, and a team that does this will chase a quality crisis that does not exist. Recovering an honest population estimate requires weighting each observation by the inverse of the probability that it was selected, so that heavily oversampled strata count proportionally less. This is mechanical, but it has to be done deliberately, and the raw queue statistics have to be labeled clearly enough that nobody quotes them as a system-level accuracy figure.

And strata defined by model-produced signals drift when the model changes. A threshold that split the population one way under one model splits it somewhere else under the next, and nothing in the policy changed to announce it. Stratum membership shifts silently, effective sampling rates move without anyone having edited a rate, and the comparison across versions quietly stops being like for like. Any signal produced by the system under test is a moving definition of the population, and it has to be re-examined whenever that system changes.

Error density changes how well the reviewing is done

Reviewer attention is not actually a fixed quantity waiting to be allocated. It is a function of what the queue contains, and this is the point at which sampling design stops being an optimization problem and becomes a question about whether the review program works at all.

The base rate of a queue does not stay a property of the queue. It becomes a property of the reviewer. Someone who has spent a month finding that almost nothing is wrong has learned, correctly, that almost nothing is wrong, and their judgment adjusts to what they have actually observed. Scrutiny falls to meet the evidence. None of this is negligence, and no amount of retraining reverses it, because a rational observer holding a strong prior behaves in exactly this way. The consequence is that the reviewer’s own accuracy degrades precisely as the queue gets cleaner. A review process whose queue mirrors a highly accurate system is a process that trains its reviewers not to look, and it will therefore miss a meaningful share of the few real errors it does surface. That effect is invisible in the metrics, because a review program that catches nothing looks identical to a review program that had nothing to catch.

Error density in the queue therefore measures something beyond the efficiency of reviewer time. It feeds back into the quality of the reviewing itself. Raising it by stratification does two things at once: it puts more real errors in front of reviewers, and it makes reviewers better at recognizing them, because the task stops being confirmation and becomes judgment.

That reasoning also implies an upper bound, which is worth stating because the logic above pushes in only one direction. A queue that is mostly wrong is not a triumph of stratification. It is a signal that the model should not be releasing output in that stratum at all, and that the humans have quietly become the pipeline rather than an audit of it. When a lane’s error rate climbs high enough that reviewers are effectively producing the output rather than checking it, the correct response is to stop routing that stratum to review and pull it out of automated release entirely, until the underlying model is fixed. Review is a control on a system that mostly works. It is not a substitute for one that does not, and stratification is the instrument that will tell you which situation you are in.

A sampling policy is a hypothesis about where the system fails

The deepest thing to understand about a review sample is that it is not a neutral window onto the output. It is a statement of belief. Uniform random sampling states that errors are distributed like the output, that every output is equally worth checking, and that nothing you already know predicts where the failures are. Stated that way, the claim is obviously false in every real system, which is what makes the uniform default so expensive: it is the one policy that refuses to use any information you have.

A stratified policy makes the opposite move. It commits to a belief about where failure lives, encoded in which strata exist and what rate each one gets, and then it spends the budget accordingly. That is a better use of what you know, and it comes with an obligation. A commitment about the world can be wrong, and a policy that concentrates attention can only find what it went looking for. The random floor is what keeps the commitment honest, the per-segment quotas are what keep the map from having blank regions, and the recurring re-derivation of the strata is what keeps the policy attached to the system as it currently behaves rather than as it once did.

Get that structure right and the review program stops producing a percentage nobody can act on and starts working as an instrument: one that concentrates real errors densely enough to diagnose, keeps reviewers sharp enough to catch them, preserves an unbiased view of true quality alongside the targeted one, and remains capable of being surprised. What you are buying with a sampling policy is not confidence that the system is fine. It is the ability to find out that it is not.