The System Prompt Is Where the Constants Live

Every prompt you send a model is really two documents fused into one. There is the part that is identical on every request in a session, and the part that is different every time. The single most consequential decision in prompt architecture is where you draw the line between them, because that line determines what the model treats as fixed law and what it treats as the passing content of one exchange. Get the placement right and behavior holds steady across thousands of varied requests. Get it wrong and you pay for it in one of two currencies, either wasted tokens on every call or rules that quietly stop being enforced halfway through a conversation.

The defining property of the system prompt is not that it is special instruction text. It is that the system prompt is resent unchanged, ahead of every user turn, for the life of the session. That one mechanical fact drives every guideline worth stating about it. Because it is constant, it is the natural home for anything that should also be constant: who the model is, what it must always do, what it must never do, and the background it needs to answer any request at all. Because it is resent every time, anything you place there that does not need to be constant is a tax you pay on every single call. The whole discipline of writing a system prompt reduces to honoring that property instead of fighting it.

The dividing line is constant against variable

The clean rule for what belongs in the system prompt is that it holds what stays the same and nothing else. Whatever is true for the entire session belongs above the line: the identity the model answers from, the behaviors it has to hold to on every reply, the shape its output is required to take, and the reference material every answer leans on. Below the line goes whatever is true for only this exchange: the question actually being asked, the records or documents attached to it, and the values that shift from one call to the next. That is the whole boundary, and it is worth naming as a boundary because both ways of crossing it carry real and asymmetric costs.

Constancy is what qualifies material for the block, but it is not sufficient on its own, and the difference matters more than it first appears. Reference data can be perfectly constant across a session and still bear on one request in twenty. Constant earns it a place above the line; being in force on every request is what earns it a place in the block that loads unconditionally. Material that is fixed but only sometimes applicable wants a third home, a layer that resolves when its condition holds, rather than riding along on every turn it has nothing to say about. Two bins are the intuitive split. The decision needs three.

Push variable data up into the system prompt and you have made per-request content permanent. A customer’s account details, the document a user just pasted, the parameters of one particular query: place any of these in the system block and they are resent and reprocessed on every subsequent turn, whether or not they still matter. The token count inflates on each call. And the stable prefix you could otherwise reuse across requests is no longer stable, so the savings it would have bought evaporate. You are paying to re-transmit the ephemeral as if it were foundational.

Push standing rules down into the user turn and you get the opposite failure, which is worse because it is silent. A rule stated only inside one user message is a rule the model saw once, competing with every message that came after it. As the conversation grows, that instruction recedes into the history, and the model weighs it against more recent turns that may pull in a different direction. The behavior you thought you had guaranteed becomes a behavior the model happens to remember, and on a long or adversarial conversation it will eventually not remember. Standing rules earn their standing precisely by living in the block that is re-presented at full strength on every turn. Move them out of it and you have converted a guarantee into a hope.

Thoroughness and cost pull in opposite directions

Even content that sits correctly above the line rides along on every request. So how much to put there becomes a genuine tension with no clean resolution. Detail buys consistency. The more of the edge cases, the formatting, and the worked targets you pin down, the less the model is left to improvise, and improvisation is where behavior scatters. But every sentence of that detail is permanent overhead. Because the block is resent on each call, a word that would earn its place in a one-off document may not earn it here. Whatever you include adds to the token count and the latency of every request for as long as the prompt stands. A lean block is cheaper and faster per call and leaves more to the model’s defaults; a heavy block is more predictable and costs more every time. Neither pole works as a policy.

The way through is to separate the two goals in time rather than trade them off in a single draft. Write first for correctness alone, putting in whatever it takes to make the model behave the way the task demands, and get that behavior stable before counting a single token. Only then optimize, cutting language that is redundant, over-specified, or merely restating something the model already does reliably, and watching whether consistency holds as each cut lands. What survives that second pass is the prompt earning its cost, the words actually doing work rather than the words that felt thorough to write. Optimizing for length in either direction before that point is guesswork, because you cannot tell a wasteful sentence from an essential one until you have watched the behavior move when it is gone.

Five jobs the block has to do

The constant block is easier to get right when you stop seeing it as one undifferentiated wall of instruction and start seeing the separate jobs packed into it. Each job covers a different kind of gap, and each leaves a different hole when it is missing. Five of them account for nearly everything a system prompt has to carry.

One job fixes identity: the vantage point the model answers from, and the register and vocabulary that come with a particular kind of expert. This one does the heaviest lifting, because it is read before everything else and colors how the model takes in every line that follows. A second supplies the standing background every answer draws on, whether domain facts, organizational policy, or fixed reference data, with anything narrower than that pushed out to a conditional layer. A third states the work to be done, the procedure to follow and the shape a correct response should take. A fourth draws the boundaries the model may not cross, the moves it must refuse and the ceilings on tone and length. The fifth makes a good output concrete by showing one, so the target is demonstrated instead of only described.

The reason to think in these five jobs is not that the labels are sacred. It is that each one, when missing, is filled by the model’s own default rather than left blank. Omit the role and the model adopts a generic one. Omit the constraints and it decides for itself what is off limits, which usually means nothing is. Omit the examples and it infers the target format from the instructions alone, which is exactly the inference you were trying to remove. A gap in the system prompt is never empty space. It is space the model fills with an assumption you did not get to make, and the value of the five-part decomposition is that it makes each of those gaps visible before the model finds it for you.

Order encodes priority

Sequence is not cosmetic, because the model reads top to bottom and whatever it reads first conditions how it reads the rest. That one fact settles most ordering questions. Identity goes at the front. When the model already knows it is, say, an auditor reviewing a filing, every instruction that follows lands differently than it would for a blank slate. Put the frame first and every later line inherits it. The work to be done should come before the boundaries around it, so there is a defined behavior for the limits to constrain rather than a fence drawn around an empty field. Demonstrations come at the end, where they illustrate rules the model has already absorbed, instead of arriving as patterns it has to distill into rules it has not yet been told.

Position matters inside a section too, for a reason rooted in how attention spreads across a long context. The material at the two ends of a long input is weighted more heavily than the material stranded in its interior, a pattern often named for the way the middle gets lost. So a rule you cannot afford to see softened, a hard prohibition or a non-negotiable format requirement, should sit near the top of its section rather than in the low-attention interior of a long block. This is less an argument for keeping prompts short than an argument for deciding deliberately what gets consigned to the weak middle. A critical rule dropped into the exact spot where the model weights it least is a rule you have written but not really installed.

Rules that both fire are the hardest bug

The nastiest defect in a constant block is not an absent rule or a wrong one. It is two rules that are each defensible on their own and quietly incompatible together, because that kind of contradiction survives the tests you are most likely to run. Suppose the block says to keep every answer to a single short paragraph, and elsewhere says to cite the governing policy clause behind every claim. On a one-fact question the two sit together fine, the happy-path check passes, and the conflict ships undetected. On a question that touches five policies they come apart: one rule demands compression, the other demands five citations. The model quietly resolves the standoff in favor of whichever it happened to weight more this time. The output is now unstable in exactly the dimension you believed you had pinned down, and the cause traces back to a collision you never watched fire on an easy input.

The repair is to make the rules conditional instead of absolute, so the case where they collide is decided in the prompt rather than left to the model. The fix is a single conditional line, something like: keep to one paragraph, unless a full answer needs more than one clause cited, in which case give the citations room. A sentence like that names the winner in advance, before the model has to invent one. The habit it points to is a pairwise audit. Read the standing rules two at a time and ask whether any pair could land on the same request and pull it in different directions. Everywhere the answer is yes, one of the two needs a condition attached, because a pair of unconditional rules that can both fire on the same input is not two instructions. It is a single decision left unmade, handed to the model to settle differently each time.

A single passing case proves nothing

A system prompt is a behavioral specification for a component that other things depend on, and it deserves the scrutiny any such specification gets, which starts with refusing to trust one passing case. What actually matters is that the behavior stays put across the many surface forms a real request arrives in, so the test set has to sweep those varied phrasings and confirm the wording of the question does not move the answer. Boundaries in particular have to be probed by inputs built to breach them, requests engineered to coax the model past its own limits. A limit that has only ever met inputs that did not press on it has not really been tested. The awkward-shaped inputs belong in the set too: the ones far longer than anything expected, the empty ones, the ones that veer to a new subject partway through a session, the shapes that reveal whether the block bends or snaps at its edges.

The part that is easiest to skip and most expensive to skip is the regression pass. A system prompt is edited far more often than it is written, and every edit that fixes one behavior can silently disturb another, because the sections interact and a change to one rule shifts how the model weights the rest. Treating each edit as something to re-verify against the cases that already passed is what keeps the prompt from drifting into a state where each individual fix was correct and the whole no longer behaves. A prompt that has never been run against inputs designed to break it is not a reliable prompt. It is an untested one whose failures simply have not been provoked yet.

The one document worth building like a specification

All of this repays the effort because of where the block sits. It governs every answer the model gives across a session, and it does so from a single place under your complete control, which makes it one of the few points in an agentic system where a small, careful change propagates cleanly to everything downstream. Honor the property that defines the block, that it is the constant part resent on every turn, and the rest falls out of it. The constants live in it and the variables live in the turn. Each of its jobs is stated rather than surrendered to a default. Its sequence puts first what should be read first, its length reflects only what is load-bearing, its rules carry conditions wherever they could collide, and its behavior is checked rather than assumed. Built that way, the block stops being instruction text that happens to work and becomes the specification the rest of the system is entitled to rely on. Little else in the design returns as much for the care.