Sampling Controls Buy Variance, Not Correctness

Every token a language model emits is drawn from a probability distribution over its vocabulary, and the sampling controls are the only place in the stack where you get to alter that draw. They are commonly treated as a quality dial: turn them down when you want the system to be right, turn them up when you want it to be interesting. That framing is wrong in a way that quietly damages production systems. These controls do not move the model toward the truth. They decide how much of the model’s own uncertainty is allowed to reach your output. What you are buying, in either direction, is variance. Correctness is not on sale at any setting.

Understanding that changes what the controls are for. They stop being a knob you fiddle with when results disappoint and become what they actually are: a deployment-level decision about how much of the distribution your system’s downstream behavior can tolerate, made per class of work, measured, and versioned alongside everything else that governs a production path.

One control reweights, two of them truncate

The three controls in common use do not differ by degree. They intervene at different points and have different mathematical characters, and conflating them is the root of most tuning confusion.

Before any token is chosen, the model has assigned every entry in its vocabulary an unnormalized score. Temperature scales those scores on their way through the function that turns them into probabilities, and the effect is a reshaping. Below 1.0, the gaps between the leading candidates and the rest widen, and probability mass concentrates on the front-runners. At 1.0, the scores pass through untouched and you are sampling from exactly what the model believes. Above that, where an interface allows it, the distribution flattens and long-shot tokens gain ground they did not earn. The critical property is that temperature never removes anything. Every token in the vocabulary retains a nonzero probability at any temperature above zero. It only changes the odds. A one-in-a-million token at low temperature is still a one-in-a-million token, and across enough generations it will eventually be drawn.

The two cutoff controls do something categorically different: they delete candidates outright. Nucleus sampling ranks the candidates, then keeps only enough of them from the top to account for a stated share of the total probability, and everything below that line is gone. The size of the surviving set is a readout of the model’s certainty. Pile most of the mass onto a couple of front-runners and the stated share is accounted for almost immediately, leaving a handful of survivors even at a permissive setting. Spread that same mass thin across many plausible continuations and the line falls much further down the list. The pool tracks the model’s own confidence, because the criterion is probability rather than position.

The other cutoff sets a hard count. Keep the top so many candidates, drop the rest, regardless of what the probabilities look like. This is cheaper to reason about and gives you a bound you can state exactly, and that bound is completely blind to the shape of the distribution it is cutting. If the model is certain and the second-ranked token already carries almost no weight, a generous count admits a long tail of tokens that have no business being candidates. If the model is legitimately torn across dozens of continuations, the same count clips options that were doing real work. It applies the same cut to a spike and to a plateau.

That is the fundamental split worth carrying: temperature reweights, the cutoffs exclude. Reweighting changes what is likely. Exclusion changes what is possible. A system that needs a guarantee that a certain class of token can never appear does not get it from any of them. Temperature excludes nothing at any nonzero value, and the cutoffs bound the size of the candidate pool rather than its contents, so a token you wanted gone is admitted at any threshold on the steps where it happens to rank high. A guarantee about which tokens may appear comes from the decoding layer, not the sampler. A system that wants nuanced control over how much of the model’s uncertainty it accepts is served better by a criterion that reads the distribution than by one that counts positions in it.

Determinism is a reproducibility property, not a correctness one

Set temperature to zero and the model takes its highest-scoring token at every step. The output becomes, for practical purposes, a function of the input. The same request produces the same answer. Teams reach for this reflexively when a system has to be trustworthy, and they are getting something real. They are just not getting what they think.

Greedy decoding returns the model’s most probable path. The most probable path is not the correct path. It is the one the model’s parameters favor, which is a statement about the model, not about the world. If the model is confidently wrong about something, temperature zero does not correct it. It locks it in. The error arrives on every single call, identically, with no variation to signal that the model was ever unsure. Sampling at a higher temperature on the same wrong belief would at least produce inconsistency, and inconsistency across repeated calls is a signal. Determinism removes the signal without removing the error.

What determinism actually buys is reproducibility, and reproducibility is genuinely valuable. It makes an evaluation suite meaningful, because a score change can be attributed to the change you made rather than to the draw. It makes a production incident debuggable, because you can replay the request and see the failure rather than hunt for it. It makes a regression test possible at all. These are strong operational reasons to pin the draw, and they are the honest case for doing so.

They are also all reasons about the system’s observability, not about the quality of any individual answer. The distinction matters because the two get conflated in exactly the situation where the stakes are highest. A team building a path that must not produce a bad answer sets temperature to zero, records that the path is now deterministic, and treats the correctness question as handled. It is not handled. It has been made repeatable. Reliability in a system that calls a model comes from validation, from constraining what the output is allowed to be, and from the ability to detect and route around a bad result. It does not come from the sampler. The sampler has no notion of what a good answer is.

Zero is a choice with failure modes of its own

Because it looks like the conservative option, temperature zero gets adopted as a default rather than chosen as a setting. It has real costs, and they are not confined to creative work.

The obvious one is the flattening of generative output. Always taking the most probable continuation produces prose that is fluent and lifeless, and it produces the same lifelessness every time. Anywhere the system’s value depends on the output not being the single most predictable string of words, greedy decoding is directly opposed to the goal. Conversational surfaces degrade in a subtler version of the same way, taking on a mechanical evenness that reads as canned because it is: the same input reliably yields the same phrasing.

The failure mode that catches production systems is different, and it is worth being explicit about because it is easy to miss during design. Retry logic frequently assumes variation. A step fails, the system tries it again, and the hope is that the second attempt goes differently. At temperature zero, with an unchanged prompt, it does not go differently, because nothing in the setup leaves room for it to. The retry reproduces the first attempt and burns a call, some latency, and a slot in the retry budget to obtain the same failure it already had. The retry loop appears to be working, in the sense that it executes, and it is functionally dead. Any recovery strategy that depends on resampling the model needs a source of variation to draw from, whether that is a nonzero temperature or a genuinely altered request. Determinism turns a whole category of recovery mechanism into ceremony.

The mirror image is equally real. Anywhere the point of the call is to survey what is possible rather than commit to one answer, variance is the product. Ask for several candidate approaches at a temperature near zero and you get one approach restated several times, because there is only one most-probable path and every draw finds it. The parameter that looks like caution has, in that context, silently defeated the request.

The right setting is a property of the task, not of the organization’s risk appetite. Extraction, classification, routing, and anything with a single defensible answer benefit from a sharp distribution, because the whole job is to commit. Work whose value lies in coverage, exploration, or voice needs the distribution left intact enough to explore. Applying one number across both because the low end feels safer is not conservatism. It is a design decision made by reflex, and it has a cost in both directions.

The order of operations hides attribution

The three controls do not act independently. They form a pipeline, and the sequence matters. By the time a cutoff runs, temperature has already had its way with the distribution, so the cutoff is reading a set of probabilities the model never produced. This has a consequence that surprises people: the same nucleus threshold means something different at different temperatures. Lowering the temperature concentrates mass on the leading tokens, so the cumulative threshold is reached sooner and the surviving set shrinks, without you touching the threshold at all. You changed one number and moved two behaviors.

A hard count interacts with temperature in a less coupled way, since a fixed position cutoff does not care how the mass was redistributed above it, but the composed effect is still not something you can predict by reasoning about each control alone. Nothing about the combination decomposes cleanly back into the parts that produced it, which means that when you move two controls at once and the output changes, you have learned nothing about which move caused it. You have one observation and two candidate explanations, and no way to separate them without redoing the experiment properly.

Hence the discipline that every serious practitioner converges on: one control moves per experiment, and temperature and the nucleus threshold are treated as alternatives rather than a pair. The advice sounds like methodological fussiness. It is not. It is the only way the exercise produces knowledge instead of anecdote. Change one number, hold everything else, measure against a fixed set of representative inputs, keep or revert on the evidence. A tuning session that ends with “we changed a few things and it seems better” has produced a configuration nobody can explain, defend, or safely modify later, which is a configuration that will be reverted in a panic the first time output quality drifts.

Variance compounds across a multi-step run

Reasoning about sampling in terms of a single call understates the effect in any system that runs a model repeatedly. A per-token probability of taking an unusual branch is small. A system that makes one model call per request feels that smallness. A system where a model plans, calls a tool, reads the result, decides the next action, and repeats does not, because each step is a fresh draw conditioned on everything the previous draws produced.

Divergence in this setting is not additive. It is multiplicative and self-reinforcing. An unusual token early in a plan does not merely add one oddity to the output. It becomes context for the next step, which is then sampled from a distribution shaped by it, and the trajectory has moved. Two runs of the same agent on the same input can end up doing genuinely different work, not because either made an error, but because the first divergent draw put them in different places and every subsequent draw compounded the gap. The variance you accepted per token, entirely reasonably, has become variance in behavior at the level of the run.

This changes the calculus of where to spend variance. In a long autonomous loop, sharper sampling on the steps that choose actions is often the right call even when the same system is happy to sample loosely on the steps that produce prose, because a strange word in a summary is a strange word while a strange choice of tool is a different execution path. It also explains a common and disorienting operational experience: an agent that works in testing and behaves unrecognizably in production on inputs that look similar. Nothing broke. The distribution was always wide enough to reach that trajectory, and enough runs eventually did. A system that samples a thousand times a day will find the tail of a distribution that a hand-run demo never touches.

The reasonable design response is not to force determinism across the loop, which brings back every failure mode above and makes recovery-by-retry impossible. It is to stop treating the variance as an accident. It is a known property of the configuration, it compounds with the number of steps, and the way you contain it is by bounding what any single step is permitted to do, not by pretending the draws will always land where they did in testing.

The defaults are a chosen position

The default sampling configuration of a serious model is itself an engineering result. Someone fitted it against a wide spread of work at once, looking for the setting where no single kind of task is served badly, and shipped the compromise that survived. It carries information: it is the setting under which the model’s behavior is best characterized and most widely exercised.

This has a practical consequence for how tuning should be sequenced. The precondition for touching a sampling control is evidence that the defaults are failing you, and evidence means measurement rather than impression. Tuning a system that does not have a distribution problem is not neutral. It is a change to a production path, made on a hunch, that moves the system away from the best-understood region of its own configuration space and into a region only you have explored. The upside in that scenario is zero, because the outputs were already acceptable, and the downside is a novel failure surface nobody else has ever tested.

When output quality is genuinely inadequate, the sampling controls are also rarely the first thing that is wrong with it. Vague instructions, missing context, a schema the model cannot satisfy, and a task decomposed badly all produce bad output, and none of them are fixed by resampling from a differently shaped distribution. Reaching for the sampling controls first is appealing precisely because it is easy: it is one number, it requires no rewriting, and it feels like engineering. It is usually treating a symptom whose cause is upstream in the prompt or the design. Sampling has a place in the sequence, and that place is after the system already works, as the last few percent rather than the repair.

Sampling settings are deployment configuration with a blast radius

The most consequential mistake with these controls is not choosing a wrong value. It is treating them as too small to govern.

A sampling parameter meets every definition of production configuration. It silently determines the behavior of every request on the path it applies to. It has cost implications, because higher variance means more retries, more validation failures, and more human review. It has an operational profile, because it decides whether an incident is reproducible. It changes the meaning of test results, since an evaluation run under one configuration does not describe a system serving under another. A number with those properties deserves the same treatment as a timeout or a connection-pool size: chosen deliberately, recorded with its rationale, versioned with the code, and changed through the same process as anything else that governs a live path.

The rationale is the part that gets dropped, and it is the part that costs the most later. A value with no recorded justification is unmodifiable. Nobody can tell whether it was tuned against evidence or typed once and forgotten, so nobody will touch it, and it hardens into a superstition that survives every rewrite of the system around it. A record of the evidence that moved the number, and of what the number was before, keeps the setting alive as a decision rather than a fossil.

There is a durability problem underneath this that makes the record matter more. A tuned sampling value is fitted to the shape of a specific model’s distribution. Swap the model, and the number is a fit to something that no longer exists. The new model’s probabilities are distributed differently, so the threshold that captured a sensible candidate set now captures a different one, and the temperature that produced acceptable commitment now produces something else. The setting does not announce that it has become stale. It keeps applying, and the output drifts for reasons the recorded configuration does not explain. Any tuned sampling value is a dependency on the model it was tuned against, and it needs to be revalidated when that dependency changes, which means the evidence it was based on has to still exist in a form someone can re-run.

The corollary for evaluation is direct. Sampling configuration belongs to the definition of the system under test. An eval suite run at one temperature and a production path served at another are measuring and running two different systems, and the gap between them is invisible: the code is identical, the prompts are identical, and the results simply do not transfer. If a variance-sensitive path is going to be judged by an eval, the eval has to sample the way production samples, or the number it produces is about a system nobody deployed.

Variance is an input you choose, not a defect you minimize

The three controls answer one question in different ways: how much of what the model considers plausible is allowed to reach the output. Temperature answers it by adjusting the odds and excluding nothing. Nucleus sampling answers it by cutting on cumulative probability, so the surviving set widens and narrows with the model’s own confidence. A fixed count answers it by cutting on rank, which is simpler, bounded, and indifferent to whether it is slicing a spike or a plateau. None of them makes the model know more than it knows. They arbitrate how much of its uncertainty you are willing to see.

Which means the real decision is never “what temperature should we use.” It is which classes of work in the system need commitment and which need range, and that question has different answers for different paths inside the same application. A system that classifies an inbound request, drafts something for a person to read, and decides which tool to invoke has three distinct variance requirements, and one global setting will serve at most one of them well. Route the paths, set each one on evidence, move one control at a time, keep the defaults where nothing is measurably wrong, and record why each deviation exists.

The framing to abandon is the one that treats variance as noise to be minimized and low temperature as the responsible choice. Variance is not noise. It is the model’s uncertainty made visible, and hiding it does not resolve it. A system that suppresses variance has not become correct; it has become confidently, repeatably, and undetectably whatever it already was. Correctness comes from what the system does with an output after the model produces it. The sampling controls decide only how much of the distribution you agreed to look at.