Speed and Durability Decide Where an Agent’s State Should Live

Every agent accumulates state as it runs: the plan it is working from, the outputs of the steps it has finished, the results of the tools it has called, the facts it retrieved to make a decision. That state has to live somewhere, and there are only two categories of somewhere. It can sit inside the context the model is already reading, or it can sit outside the model in a store the agent reaches through a tool call. Most systems never make this choice deliberately. They let state pile up wherever it first appeared, which usually means everything stays in context until the window overflows, or, in the opposite reflex, everything gets written to a database because durability felt responsible. Both defaults are wrong for most of the state they apply to, because the choice is not a matter of taste. It is governed by two properties of the state itself: how fast it has to be read, and how long it has to survive. Those two properties pull in opposite directions, and where a given piece of state belongs is the point where you resolve that tension for that piece specifically.

The reason this matters is that the two placements do not degrade gracefully into each other. In-context state that needed to be durable does not survive an interruption, and there is no warning when it vanishes. External state that only needed to be fast turns every read into a network round trip and every write into a new way for the turn to fail. Getting the placement wrong does not produce a slightly worse system. It produces a system that loses work it thought it had, or one that is slow and brittle for no benefit. The placement decision is small in isolation and compounding in aggregate, because an agent makes it dozens of times per run whether or not anyone designed it to.

In-context state is fast because the model is already reading it

In-context state is whatever the model can see while generating a token: the running conversation, whatever the tools it invoked this turn returned, any document that got injected into the prompt, and the standing instructions it works under. What makes this placement fast is not a fast fetch but the absence of one. The model never has to go get this state; it is already in front of the model, part of the same forward pass that produces the next token. Read latency is zero because there is no read, only presence. For anything the agent leans on continuously while it works, that property is decisive. State consulted on every reasoning step cannot afford a round trip on every reasoning step, and held in context it never incurs one.

That speed is real, but calling it “free” oversells it, and the oversell is where the trouble starts. Reading in-context state is free. Keeping it there is not. Each turn re-submits the entire window to the model and pays to process it again, so state that lives in context is not bought once; it is rented, in tokens and in the model’s finite attention, for every turn it remains. A fact that mattered ten steps ago and will never matter again still consumes budget and still competes for the model’s focus with the facts that matter now. In-context state is cheap to read and expensive to keep, which inverts the usual intuition, where holding onto a value is the cheap part and looking it up is what costs.

The harder limit is durability, and the danger is that its failure can pass unremarked. The context window is bounded and fixed, and once a task’s accumulated state reaches that bound, something has to give. The platform will not quietly swallow the overflow: it either rejects the request or halts generation at the limit. What happens next is the application’s call, and one common way to keep the agent running is to shed the oldest turns and continue. That choice is where state disappears without a signal. Nothing in the model’s view flags the omission, so the agent carries on reasoning over a version of the task that has quietly lost its earliest pieces, having decided, at the application layer, that losing them beat stopping. Session end is the blunter version of the same problem: in-context state does not degrade or thin out, it is gone entirely, unless something committed it elsewhere first. This is what disqualifies context as storage regardless of how large the window becomes. Growing the window lets more state stay hot at once and delays the point where the application has to start shedding, both real gains on speed and cost. Neither touches durability. A window of any size still retains nothing once the session closes, so treating context as the home for durable state is a category error that no amount of window keeps you from making.

External memory buys durability with latency and a new failure surface

External memory lives outside the model, in systems the agent touches only through tool calls and that go on existing no matter what happens to any one context: key-value stores, relational databases, vector indexes. Its defining advantage is the one thing context cannot provide. It persists. External state survives truncation, survives the session ending, survives the process crashing partway through a task. A run that dies does not take the written-out state with it. External stores are also queryable in ways an undifferentiated context is not: you can look state up by an exact identifier, or filter and join across structured fields, or retrieve by meaning. State that has to be found later, rather than merely held now, wants to live somewhere it can be queried.

What you pay for that persistence is latency, and the charge is not incidental. Reaching external state means issuing a tool call, and every such call, read or write alike, is a round trip that the in-context alternative simply never takes. This is the exact inverse of the in-context tradeoff, and the symmetry is the whole point: context gives you speed and denies you persistence, external memory gives you persistence and charges you latency. Neither is a strictly better version of the other, so there is no default that is right in general, only a placement that is right for a given piece of state given what that state has to do.

Latency is the obvious cost. The subtler one is that externalizing state adds a failure surface that in-context state does not have. A value held in context cannot fail to be read; it is simply present. A value held externally is fetched through a call that can time out, return stale data, or fail outright, and the agent now has to handle the case where the state it depends on is temporarily unreachable. There is also a consistency hazard whenever the same fact exists in both places at once. If a piece of state is cached in context and also persisted externally, the two copies can drift, and the system needs a clear answer to which one is authoritative when they disagree. Duplicating state across the boundary without deciding that question is how an agent ends up acting on the stale copy while the correct value sits in the store it never re-read.

Choosing a store is committing to an access pattern

External memory is not one thing. It is a small set of store types with genuinely different query models, and choosing among them is choosing how the agent will later reach the state. Getting this wrong does not usually cause an outright failure; it causes a system that works but fights its own storage on every access.

A key-value store answers one question well: given an exact identifier, return the value. It is the simplest option and the fastest for direct lookup, and it fits state that has a natural key and is retrieved by that key. Current task status, a user’s stored profile, a configuration record: each is something you ask for by name, and a key-value store gives it to you without ceremony. A relational store answers a different class of question. When state has structure and the agent needs to filter, join, or query across several fields, a relational model is what supports that without forcing the agent to load everything and sort it out in context. Records, logs, and anything that has to be examined along more than one dimension belong here. A vector index answers the question the other two cannot: retrieval by meaning rather than by key. When what the agent needs is the note or passage that best fits the situation it is in, and no exact identifier can express what “best fits” means, semantic similarity over embeddings is the access pattern that answers it.

The failure is the mismatch. Reaching for a key-value store when the agent actually needs to retrieve by meaning forces you to invent keys for a lookup that was never keyed, and the retrieval degrades because the store’s query model does not match the access pattern. The rule is to match the store to how the state is genuinely reached, not to how it is most conveniently written. Semantic retrieval deserves one caution of its own, because its convenience hides a property the other two do not share. Key lookup is exact: you get the value under that key or you get nothing, and there is no ambiguity about whether it matched. Similarity search is approximate by construction. It returns the nearest neighbors whether or not any of them is actually the right answer, so it can hand back a result that sits close in embedding space and is nonetheless wrong on the merits. That lossiness is acceptable, often ideal, for retrieving context to inform a decision. It is the wrong mechanism for state that has to be recalled exactly, and using it there trades an occasional silent error for the convenience of not having to name a key.

The default is context, and the best storage is often none

Because the two placements trade opposite costs, the useful way to make the decision is to classify the state by whether it has to survive beyond the current context at all. Most state does not, and the default should reflect that. State that the current turn is mutating and touching over and over is hot, and hot state belongs in context, where the model reads it at zero latency. Reaching for external storage for hot state is a straightforward mistake: it pays the latency cost on every access while adding nothing, because the state was never going to outlive the context it is being used in. Speed is the reason to keep it close, and there is no competing durability requirement to override that.

State earns a move to external storage when it has to persist past the point where context can hold it, and only then. State that is rarely touched during active work but must be available in a later session is cold, and cold state is exactly what external stores exist for. The escalation from in-context to external should be driven by a specific durability requirement the context genuinely cannot meet, not by a general sense that persisting things is safer. Over-persisting is not a harmless excess. It buys latency and a failure surface with tokens that had no return, because state that did not need to survive gained nothing from being written down.

The category that gets missed is the one where the right amount of storage is none. Some state is reconstructible: it can be regenerated from source data or recomputed by calling a tool again, at a cost low enough that storing it is not worth the trouble. For that state, the storage decision is often to store nothing and rebuild it if it is ever needed, because a copy that can be cheaply reproduced is a copy that can also silently go stale, and keeping it introduces a consistency problem to solve a durability problem that did not exist. The instinct to save everything defensively works against you here. The cheapest durable state is the state you never had to persist because you could always make it again.

A few tests settle most cases at design time, and they are worth applying in order of consequence. The one that forces the issue is recoverability: if the session were to end at this instant and the state could not be rebuilt from anything outside the context, it has to be written down, and there is nothing further to weigh. Everything after that only refines where and whether. State reached by resemblance rather than by a known identifier has its home in a vector index, for the reasons store selection already lays out. State that anything other than this same context will ever read, a later session, a sibling agent working now, a different user entirely, is persistent by definition and not by choice, because separate contexts hold nothing in common and an external store is the only ground where they can meet. And running the other way, state that can be regenerated cheaply from its source whenever it is wanted may deserve no durable copy at all. Applied together, these tests keep a system clear of both failures at once: never losing the state whose loss cannot be undone, and never paying to persist the state that only ever needed to be fast.

State does not stay in one category

The classification is real but it is not static, and treating it as a one-time label is its own source of error. State changes temperature as the task moves. The output of a step is hot while the next step is consuming it and cold the moment that consumer is done, at which point it becomes either history worth persisting or noise worth dropping. A plan is hot for the whole run and cold, or discardable, once the run completes. Good state management is not a single placement decision made up front. It is the ongoing work of promoting state into context when it becomes relevant and demoting it out when it stops being, so the hot tier stays small and current instead of silting up with results that have already been used.

This is what a two-tier design does in practice, and it is what most production agents converge on, not because hybrid is fashionable but because neither placement alone fits a real workload. The context is the hot tier, holding only what the run is using right now: the plan in force, the current step’s inputs and its results, the tool output this turn actually consults. The model reads all of it at zero latency, and none of it is paying a round trip it does not have to. The cold tier is external, holding what must survive without needing to be instantly on hand: finished results, configuration, the accumulated record of the run. Resuming after an interruption then does not mean replaying the whole history back into context. The agent pulls only the slice of cold state it needs to continue from where it stopped and leaves the remainder at rest in the store. That selective rehydration is the entire payoff of separating the tiers. It buys the speed of in-context access for the state being used right now and the durability of external storage for the state that has to outlast the moment, and it declines to pay the latency of externalizing anything that was only ever going to live and die inside a single context.

The question to ask before the storage question

Where an agent’s state should live is not a question about databases. It is a question about the state, asked before any store is chosen: how fast does this have to be read, and how long does it have to survive? The two axes are independent, and every piece of state sits somewhere on both. State that is read constantly and needed only for the moment belongs in context and nowhere else, and paying to persist it is waste. State that is read rarely but must outlive the session belongs in external memory, chosen to match how it will be queried, and keeping it in context is a loss waiting to happen at the next truncation or the next session boundary. State that can be regenerated on demand often belongs nowhere at all. The default is context because speed is the common case and durability is the exception, and the discipline is to escalate to storage only when a real requirement forces it, then to keep moving state across the boundary as its temperature changes rather than freezing it in place. An agent that treats its context as durable will lose work without noticing. An agent that treats every fact as worth persisting will be slow and fragile in exchange for a safety it did not need. The system that holds up is the one that asks, for each piece of state, how fast and how long, and puts it exactly where those two answers point.