A Failure Belongs at the Altitude of the Decision It Invalidates

Every failure in a running system is handled somewhere. It is swallowed by a component, absorbed by a retry, folded into a default value, or allowed to travel upward until something decides what it means. There is no such thing as an unhandled failure in production. There is only a failure whose handling location nobody chose.

In most systems that location was never designed. It settled. It is wherever the first error handler happened to sit, or wherever a past incident pushed it after someone got paged for noise. The result is an error-handling altitude that is the residue of the system’s history rather than a property of its architecture, and in a system where components make decisions instead of merely executing them, that residue is expensive.

The criterion is simple to state and demanding to apply. A failure belongs at the level that owns the decision the failure invalidates. Not the level that noticed it. Not the level that can technically make the error go away. Not the level whose author happened to be on call the last time it fired.

Containment and propagation fail in different directions, and not equally

Set the two extremes side by side, because nearly every real system is drifting toward one of them.

A component that contains everything catches each failure internally and returns to its caller as though the work completed. The caller keeps planning. Its next step is built on an output that may be partially wrong, quietly truncated, or invented outright. The defect does not disappear; it changes form, from a failure the system could have acted on into a fact the system now believes. It surfaces later, several steps downstream, in a place with no causal connection to its origin, which is the most expensive place a defect can surface.

A component that contains nothing sends every hiccup upward. A transient timeout that resolved on its own, a malformed field it corrected without difficulty, a rate limit it waited out. The caller now spends its attention adjudicating events that did not change anything, and worse, the failures that genuinely required a decision arrive in the same channel as the ones that did not. Signal is not lost because it was absent. It is lost because it was buried in things that looked exactly like it.

Both are bad and it is tempting to leave it there, balanced. That symmetry is false, and the asymmetry is the practical guide when a case is genuinely unclear. Over-propagation wastes attention and is visible while it does so. Someone will complain about the noise, and the complaint carries the information needed to fix it. Over-containment destroys evidence, and it is invisible by construction. Nobody complains, because the system reports success. One failure mode announces itself and can be corrected from what it says. The other is silent, and correcting it requires first discovering that it exists.

That is what licenses the default: when it is unclear whether a failure should rise, let it rise, labeled as low severity. Surfacing something the caller did not need is a mistake with a cheap remedy, a filter rule. Containing something it did need leaves nothing behind to remedy.

A component may only contain what it owns

The clean version of the rule turns on ownership, and it explains why the answer differs so sharply between components in the same system.

A thin tool adapter owns nothing. It exposes a capability and has no plan, no goal, and no authority to decide that a different approach would be better. Almost everything it encounters should travel upward, because almost everything it encounters invalidates a decision it does not own. The narrow exception is the class of failures that are purely artifacts of the transport rather than facts about the world, a connection reset before the request was seen, a retryable status on an idempotent read. Those are noise in the literal sense: they carry no information about the request’s outcome.

A delegated agent owns a subgoal. Its containment budget is exactly the set of failures that do not change that subgoal or anything its caller believes about the world. It can correct a malformed input and proceed. It can retry a call that timed out and got a clean result the second time. It cannot decide that the subgoal was misconceived, because it was not the component that conceived it.

An orchestrator owns the plan. It can reroute, substitute, drop a branch, or accept a partial result, because the plan is the thing it is authorized to change. It cannot decide that the goal itself is not worth pursuing, or that a policy should be relaxed for this request, because it does not own those.

Above that sits a level that owns the policy, the budget, and the risk, and in most systems that level is a person.

The uncomfortable structural fact underneath this is that information and authority sit at opposite ends of the stack. The component at the bottom knows precisely what happened, has the exact error text, the arguments it sent, the state it was in. It has almost no authority to decide what that means for the goal. The level with the authority to change course has no direct knowledge of what went wrong; it sees whatever the layers below chose to tell it. Propagation is the mechanism that marries the two, and its quality determines whether the authority is exercised on facts or on a rumor.

Read the rule against that asymmetry and it sharpens into a warning. When a component handles a failure it does not own, it is not being resilient. It is making a decision that belongs to someone else and declining to mention that it made one.

The propagation test is about what the caller believes, not about severity

The familiar version of this test asks whether the caller would plan differently had it known. That is a good start and slightly too weak. It invites the component to speculate about a plan it cannot see. The stricter and more mechanical form is this: after your local recovery, is everything the caller believes about the world still true?

If the answer is yes, contain it. A retried read that eventually returned the correct value leaves the caller’s model of the world intact. The caller believes it received the record, and it did. The intermediate failure is a fact about infrastructure, not a fact about the request, and the caller does not need it to plan correctly.

If the answer is no, it must rise, and the severity of the underlying event has nothing to do with it. This is where the severity framing misleads people. A trivial-looking failure that leaves the caller with a false belief is a propagation case. A dramatic-looking failure that leaves the caller with an accurate one is not.

The clearest instance is a partial side effect. A component executes three steps of a five-step operation, fails on the fourth, cleans up what it can, and reports that it could not complete. Fine. But if it reports that it did not complete without saying that three steps already ran, the caller’s belief about the world is now wrong in the most dangerous possible way. It believes nothing happened. It will retry, and the three steps will happen again. The failure was propagated and the state was not, which is the same defect as silent containment wearing an error message.

The same test resolves the authorization case without any appeal to severity. A denied permission is not a large error or a small one; it is a fact that invalidates the caller’s belief that this path exists at all. No retry will change it. No local cleverness will route around it. It rises, always, because the plan that produced the call was built on a premise that turned out to be false.

Recovery restores the contract, substitution quietly rewrites it

Containment is only legitimate when the component actually recovered. In practice a large fraction of what gets called local recovery is something else: the component could not do the work, so it produced a substitute and returned it under the same contract as a real result. A cached record stands in for a live one that was unreachable, and nothing on it says so.

Recovery restores the contract. After it, the component’s output means exactly what its interface promised it would mean. Substitution issues a weaker contract under the old name, and the ownership rule is what makes that indefensible. The contract belongs to the caller, not to the component that implements it. A component may decide how to do its work. It may not decide, on its own authority and without saying so, what its output is now worth.

That is the line between a fallback and a containment decision. A fallback that produces a genuinely weaker result is often the right answer, and containing the error that triggered it is often right too, because the caller has no use for the news that a primary path failed. The degradation is different. It changes what the caller is entitled to do with the result, which makes it a fact about the caller’s own decision rather than about the component’s internals. The error can stay local. The change in the contract cannot.

Local recovery also has to be bounded, and the reason is about altitude rather than cost. A budget, a count of attempts or a span of time, is what fixes the moment at which a contained failure becomes a surfaced one. Exhausting it is not a failure of the policy. It is the policy, executing the one transition it exists to define. Without a budget a component has no boundary at all, only an intention, and the failure it was meant to handle turns into a caller waiting on a decision that will never be made.

Containment decides control flow, never visibility

The reason so many teams end up at one of the extremes is that they are answering a question with only two options, and it is the wrong question. Propagate or swallow is a false pair. Propagation and observability are separate channels, and a failure can be, and usually should be, contained in one and emitted in the other.

Propagation is a control-flow decision. It answers whether the caller must change what it does next. Emission is an observability decision. It answers whether anyone needs to know this happened, ever, for any purpose.

Almost everything that is correctly contained should still be emitted. The retry that succeeded on the second attempt changed nothing about the caller’s plan and belongs nowhere in its control flow. It belongs absolutely in a record an operator can aggregate, because the aggregate is where its meaning lives. A single retried call is noise. A component retrying on most calls is a dependency degrading in real time, and it will be visible in cost and latency long before it is visible as an incident. That signal exists only if every contained failure left a trace an operator can count, enough to tell one kind of recovery from another and to see how often each is firing.

Contained means the caller does not need to act. It never means nobody needs to know. A local recovery that leaves no trace is not containment. It performs the same erasure as silent containment, one layer earlier, because it destroys the only evidence that the system is running in a degraded regime.

Propagating to a model costs more than propagating to code

There is a hazard here that has no analogue in conventional error handling, and it complicates the advice to surface things.

In an ordinary stack, propagating an error is cheap and deterministic. The frames unwind, a handler matches, and the same input produces the same route every time. Surfacing a failure to a caller that is a language model is a different act entirely. It is not a control-flow transfer; it is a request for judgment, delivered into a context window, answered probabilistically.

The consequences follow directly. Surfacing an error to a reasoning caller does not mean the plan adapts. It means the plan might adapt, or the model might reissue the identical call that just failed, or improvise a workaround nobody sanctioned, or write an apology and continue as if the step had succeeded. Raw error text also has a cost that an exception does not: it consumes context, and it seeds the reasoning that follows with the vocabulary of the failure. A model that reads a stack trace tends to start reasoning about stack traces.

So the altitude question has a second axis, and it is the one the classical framing misses. Choosing a level is not enough; you also choose the substrate that handles the failure at that level, code or model. Those are not the same handler, and they are good at opposite things.

A failure with a known, deterministic response should be handled in code, at whatever level owns it. If a schema mismatch has a defined coercion and an authorization denial has a defined route to a human, handing either one to an orchestrating model is asking a probabilistic component to do a job a lookup table does better, and paying context for the privilege. Only failures that genuinely require judgment, where the right response depends on the goal, the tradeoffs, and information no rule anticipated, are worth the model’s attention.

This reframes the surface-it-upward advice into something more precise. Surface it to something that can act on it. If the level above cannot do anything with the failure other than retry the call that just failed, propagating to it does not produce adaptation. It produces a loop, and the loop is expensive in a way the original failure was not.

Fidelity decays at every hop

In a system with real depth, a failure at a leaf may travel through several levels before it reaches anyone who can act. Each level does the right thing locally, which is to interpret the failure in its own terms, and the aggregate outcome is that the original fact is ground away.

The pattern is easy to recognize once named. A leaf reports that a specific record was not found. Its caller, unable to complete its subgoal, reports that enrichment failed. That caller’s caller reports that the pipeline stage did not produce output. At the top, an operator reads that the task failed, and every fact that would have made the failure diagnosable was discarded by a well-intentioned component translating it into local vocabulary.

Propagation is re-classification, not re-authoring. Each level is entitled to add its own interpretation, its own severity, its own judgment about what this means for the work it owns. No level is entitled to replace the originating fact, and the originating fact has to reach the top intact, no matter how many hands it passes through. When each hop is allowed to summarize its child’s error rather than carry it, depth works against the operator. Each additional level is one more chance for the fact that would have made the failure diagnosable to be replaced by a local restatement of it, and every one of those restatements is faithful.

Some failures have no owner inside the system

Follow the ownership rule up far enough and it produces a case people rarely design for. Some failures invalidate a decision that no level of the system is authorized to make.

A policy would have to be relaxed for the work to continue. A spend exceeds what anyone inside the run is authorized to commit. An irreversible action turns on a judgment the system does not have the information to make. For these the correct altitude is not the top of the system. It is outside it.

The failure mode is not that these get propagated too far. It is that they reach the top and get absorbed there, because the top is a component like any other and will do something with what it receives. In an agent system the top is very often a model, which makes this considerably worse: a model that receives an unroutable failure will not stop. It will produce a response, because that is what it does, and the response will be an approximation of a decision nobody authorized it to make. The system will then report a successful run.

Absorption at the top is the most dangerous form of over-containment precisely because it is the least visible. Every lower boundary was crossed correctly. The failure rose exactly as designed. And then it was quietly consumed by the last component in the chain, and the escalation that should have happened never did.

The boundary is an interface, and almost nobody tests it

The propagation boundary is not something the code discovers at runtime. It is a term in the component’s contract, and it can be stated in the same breath as the inputs and outputs: this is what I contain, this is what I surface, this is my budget before a contained failure becomes a surfaced one, and this is the shape it arrives in.

Written down, it becomes testable, and this is where the gap between systems that hold up and systems that do not is widest. Nearly every component has tests for its success interface. Almost none have tests for its failure interface. Fault injection at the boundary is the direct remedy: assert that an authorization denial reaches the caller, that a transient timeout does not, that the third failed attempt exhausts the budget and changes altitude, that a partial side effect is reported even when the operation is reported as failed. Those assertions are as cheap to write as any other test and they protect the property that is hardest to recover once it has drifted.

Left untested, the boundary is not enforced by anything. It exists in someone’s head, and it lasts exactly as long as that person’s memory of why it was drawn where it was.

Altitude ratchets downward unless someone defends it

There is a dynamic in maintenance that pushes every system toward silence, and it explains why the boundary is nearly always found in the wrong place rather than merely in an arbitrary one.

When a failure propagates too far, the system is noisy. Someone is paged for something that did not require them, the orchestrator burns spend adjudicating events it cannot act on, and the noise is felt by real people who will do something about it. The something is almost always the same: catch it lower. Add a handler in the component. The noise stops, the fix is small and clearly correct, and it is never revisited.

When a failure is contained that should have propagated, nothing happens. There is no page, no complaint, no visible cost. The system reports success. The bug is not filed, because from every observable angle there is no bug. Whatever it eventually costs is paid much later and charged to something else, and nobody goes looking for the handler that swallowed the truth, because the handler left nothing to find.

The two forces are wildly unequal. One generates constant, legible pressure to lower the altitude. The other generates none at all. The boundary therefore ratchets in one direction, toward containment, toward silence, one entirely reasonable local fix at a time, and a system that started with a defensible boundary will not have one in a year unless someone is defending it on purpose.

Defending it looks unglamorous. The boundary is stated in the contract, so a change to it is a change to an interface rather than an implementation detail. The tests fail when a new handler quietly lowers the altitude. And the review that follows an incident asks not only which failure reached too far, but which handler was in a position to know and said nothing.

The altitude is chosen once or discovered during an incident

Every system has an error-handling altitude, whether or not anyone decided on it. The only question is whether it was chosen deliberately, at design time, against the ownership of the decisions in the system, or whether it accreted from the sequence of incidents that happened to be loud.

Decide it deliberately and the rule is compact. A component may contain only the failures that leave its caller’s beliefs intact, and only within a bounded budget, and only when it has genuinely restored the contract rather than substituted a weaker one. Everything else rises, to the level that owns the decision it invalidated, carrying the originating fact undamaged, handled by code where the response is known and by a model only where judgment is genuinely required. Everything contained still gets recorded, because control flow and visibility are different questions and only one of them is being answered by containment.

Decide it by default and you get the boundary the ratchet produces, which is a system that has learned, one small reasonable fix at a time, to report that everything is fine.