A Long Conversation Can Fit in the Window and Still Lose the Thread

A conversation that runs long fails in two different ways, and they are easy to confuse because both show up as the model getting worse over time. The first failure is mechanical and visible: the accumulated history grows until it no longer fits in the window that holds it, and something has to give. The second is quieter and, in most systems, more damaging: even while everything still fits, the model drifts away from the goal it started with as new topics pile up on top of the original one. One failure is about running out of room. The other is about losing focus with room to spare. They have different causes, they surface at different points in a session, and the techniques that fix one do nothing for the other.

Treating a long conversation as a single problem, usually the capacity one because it is the one that throws an error, is how systems end up with careful token management and a model that has quietly forgotten what the user asked for. The capacity problem is real and worth solving well. But it is the obvious half. The half that erodes answer quality and surfaces as no error at all is coherence, and it is the reason a session can be comfortably inside its limits and still be off the rails.

Capacity is the failure that surfaces as an error

Turn by turn the accumulated exchanges grow heavier, and the token count moves steadily toward whatever ceiling the model imposes. This part is unavoidable arithmetic. Every exchange adds to the running history, and unless something removes tokens as fast as they arrive, the total only goes up. What happens at the ceiling is worth being precise about, because a common mental model of it is wrong. The system underneath does not, as a policy, evict your oldest exchanges to buy space. Everything said so far stays intact, and the overflow is surfaced rather than hidden: once the total exceeds what the model accepts, the request is either refused outright or generation halts at the limit, depending on the model. Rolling deletion of old turns is a behavior some chat front-ends add on top; it is not something the underlying interface does for you. Keeping the conversation inside its window is the system’s job, not the platform’s.

That responsibility is what compaction exists to discharge. Compaction buys back running room by shrinking the footprint the older exchanges occupy, so the token total rises more gently and the ceiling arrives late or never. It comes in a few shapes, distinguished by how aggressively they trade detail for space. Full summarization collapses everything that came before into a single condensed pass. It frees the most space and discards the most, treating every earlier turn as equally expendable. Selective summarization is more discriminating: it holds onto the exchanges that settled something, a decision or a result, and lets the connective tissue and dead-end reasoning go. It is the reasonable default for most conversations because it targets exactly the low-value bulk. The sliding window fixes a boundary in time, holding the newest stretch of the conversation word for word and compressing only what precedes it, which preserves precise access to what was just said while still capping the total. Recency is where a conversation’s next move usually lives, so paying for exact recent turns and compressing the distant ones matches how the model actually uses its history.

None of this necessarily has to be built by hand. The platform layer increasingly offers automatic condensation of earlier turns, clearing of stale intermediate results, and signals that let the model track how much of its budget remains. Those are worth using where they exist. But they are instantiations of the same operation, and they carry the same cost as a hand-rolled version, because the cost is intrinsic to what compaction is rather than to who implements it.

Compaction spends detail to buy room, and the bill is not always payable

Every form of compaction is lossy by construction. Turning many turns into a shorter representation means deciding what does not make the cut, and whatever gets cut is gone from what the model can see. This is not a defect in a particular summarizer that a better one would eliminate. It is the definition of the operation. A summary that lost nothing would be the same length as the original and would have reclaimed no space. The compression is the point, and the loss is the compression.

Most of the time that loss is acceptable, because most of what a long conversation accumulates is genuinely disposable. The problem is the case where it is not, and the failure there is severe and specific: applying compaction to a conversation whose value depends on verbatim history. A review that unfolds across many exchanges is the sharpest instance. If the exact contents of an earlier artifact are compressed into a paraphrase, the model can no longer quote or reason over that artifact precisely, and every later step that referenced it is now working from a description of something it can no longer see. Conversations with a compliance or record-keeping character have the same property for a different reason: the exact wording is the asset, and a faithful gist of it is worthless. In these cases lossy compression is not a tuning mistake, it is the wrong operation entirely.

The design question that sorts one case from the other is not how long the conversation is or how close it is to the ceiling. It is whether anything downstream would need to reproduce an early specific exactly, rather than merely know it. When a later step has to quote an artifact, audit its wording, or reason over its precise contents, that conversation cannot be compacted where those details live, and the capacity problem has to be solved some other way. When the general thread is what carries forward, compaction is appropriate and its loss costs nothing that matters. Open-ended research sessions, most support flows, and assistant-style interactions fall on this side: they need topical continuity, not the ability to re-quote a specific earlier output word for word. The decision is a property of the conversation’s semantics, and it should be made deliberately rather than discovered when a compacted detail turns out to have been load-bearing.

For conversations that outgrow any window entirely, the response is to stop keeping everything in the active history at all and move summaries, established facts, and durable preferences into an external store that is retrieved from as needed. That crosses into a distinct concern, persistence beyond the life of a single conversation, with its own retrieval machinery and its own tradeoffs, and it is worth treating on its own terms. Within the scope of one long-running conversation, compaction is the primary lever, and matching its aggressiveness to what the conversation can afford to forget is the whole of the capacity discipline.

Drift is the failure that happens with room to spare

Everything so far assumes the binding constraint is space. It often is not. A conversation can sit well within its window for its entire life and still degrade, because length attacks focus independently of capacity. As turns accumulate, the original goal becomes a smaller and smaller fraction of what the model is looking at. Recent turns are more numerous, more vivid, and closer to the point of generation, and the model’s attention follows the mass of the recent material. The objective the user stated at the start does not get deleted. It gets diluted. It is still in the history, technically visible, but surrounded by so much subsequent detail that it stops exerting force on the output.

The result is drift, and it is more insidious than overflow precisely because nothing breaks. There is no error, no ceiling, no signal that anything is wrong. The model keeps producing fluent, locally reasonable responses. Each individual turn looks fine. What degrades is the relationship between the turns and the original intent: constraints stated early stop being honored, scope the user explicitly excluded creeps back in, and the conversation gradually reshapes itself around whatever was discussed most recently rather than what it was for. In a multi-step workflow this is the dominant risk, because the whole point of such a workflow is to hold a distant objective steady across many intermediate steps, and drift is exactly the erosion of that steadiness. A support agent slowly stops solving the problem the customer opened with and starts chasing the most recent tangent. An assistant working a long task optimizes the last thing it was told at the expense of the task it was given.

Drift and overflow can occur together, but they are not the same failure and they do not respond to the same fix. Compaction addresses overflow by removing tokens, and removing tokens does nothing to keep a goal salient. In fact naive compaction can make drift worse, because a summary that compresses the original objective into one clause among many, then keeps the recent turns verbatim, has just increased the ratio of recent detail to founding intent. Solving capacity does not solve coherence. A conversation that has been carefully kept inside its window can be more prone to losing its way, not less, if the management strategy privileged recency without preserving the anchor.

Holding the thread means re-establishing the anchor every turn

Because drift comes from dilution rather than deletion, the countermeasure is to keep re-concentrating the model’s attention on what matters, turn after turn, rather than trusting a goal stated once at the start to hold on its own. A few techniques do this, and they differ mainly in how much structure they impose.

The lightest reprints the founding objective at the top of every turn, so the goal sits among the freshest and most weighted things in view rather than the most buried. It costs a sentence or two of tokens and works directly against the dilution mechanism, which is why, for a workflow whose characteristic failure is scope quietly widening, that single move often outperforms anything heavier. A step up from that is to open each turn with a brief recap of the relevant prior context, grounding the model in where things stand before it takes the next step, rather than leaving it to reconstruct that standing from the full accumulated history. The heaviest and most durable technique is to maintain an explicit structured record of the conversation’s state, the current goal, the facts that have been established, the decisions that have been committed, and to carry that record into the prompt on every turn. This turns the anchor from prose the model has to re-derive into a stable object it is handed directly, and because it is structured it can be updated deterministically as the conversation progresses rather than being re-summarized and re-diluted each time.

These are not alternatives to compaction; they operate on the other axis. Compaction decides what of the past to keep. Anchoring decides what of the purpose to re-assert. A robust long-conversation system usually does both, because it faces both failures: it compresses the disposable bulk to stay inside the window, and it re-establishes the goal each turn to stay on target. The presence of one is not evidence that the other is handled. They are separate obligations that happen to share a session.

The right strategy is the one the conversation’s shape requires

The two axes together define the design space, and where a given system sits in that space is dictated by what kind of conversation it hosts. This is the decision that should be made first, because it determines which failures the system actually faces and therefore which machinery it needs.

Bounded, single-purpose exchanges that resolve within one sitting need coherence far more than capacity. A well-scoped support interaction rarely threatens the window; its risk is that a long back-and-forth loses sight of the customer’s original issue. For it, in-context history plus goal anchoring is usually the whole solution, and elaborate compaction is machinery for a problem it does not have. At the other extreme, an interaction whose entire value is verbatim access to specific artifacts, a review, an audit, anything where exactness is the asset, cannot use lossy compaction where those artifacts live, and has to solve capacity by keeping the critical material exact and compressing only around it. In the middle sit the systems that need both: a long agentic task or a copilot that requires precise access to recent, exact material while the older surrounding context can be safely compressed, so selective or sliding-window compaction handles the bulk while the exact recent material and a re-asserted goal keep it both in-budget and on-target.

The principle underneath these cases is that the memory and coherence strategy should follow from what a given conversation has to do to work, not from whichever mechanism is cheapest to attach after the fact. The two questions that determine it are concrete: does any later step depend on an early detail being preserved exactly, and how far can this conversation run before the original goal stops shaping the output. The first answer sets how aggressively the history can be compressed and where it cannot be. The second sets how hard the goal has to be re-anchored and how often. A system that answers both up front builds the right amount of machinery for its actual shape. A system that answers neither tends to ship with whatever handles the error it saw first, which is almost always overflow, and to carry an unmanaged coherence problem into production where it degrades answers quietly for the entire life of every long session.

Length is not one adversary. It is two, arriving on different schedules and demanding different defenses. Capacity is the loud one, and compaction answers it, at the cost of a detail loss that has to be matched to what the conversation can afford to forget. Focus is the quiet one, and re-anchoring answers it, by refusing to let a goal stated once dissolve under the weight of everything said since. A conversation is being managed well only when both are handled on purpose. Handle only the one that throws an error, and the model will keep fitting comfortably inside its window while slowly forgetting what it was there to do.