When a Tool Needs to Think, It Should Ask the Runtime Rather Than Call a Model Itself

Sooner or later a component in your tool layer needs judgment. A document tool fetches a fifty-page contract and has to hand back something that fits in the caller’s context budget. A retrieval tool pulls a hundred rows and has to decide which eight are relevant to the question that was asked. A ticketing integration receives a free-text complaint and has to route it. In each case the component is not merely executing. It has to interpret, and interpretation is what models are for.

The obvious move at that moment is to give the component a model client and an API key. It is a small change, it works immediately, and it is one of the most consequential architectural mistakes available in an agent system. Every credential you hand out is a new place your data can leave, a new line item nobody is watching, and a new model decision made outside every policy you wrote. The alternative is to invert the call: the component does not hold model access, it requests inference from the runtime that already has it, and the result comes back through the same channel the component was already speaking on.

This is not a novel idea so much as an old instinct pointed at a new resource. A plugin does not open its own database connection; it asks the host for one. A library does not instantiate its own credentials store. Inference is simply an unusually expensive and unusually dangerous thing to hold, which makes the instinct more urgent here, not less. MCP makes this a first-class capability: a tool server can ask the client to run an inference for it, and the client decides what actually happens. The mechanism matters far less than the shape, because you will need that shape whether or not you build on that protocol.

The tool boundary was drawn on the assumption that only one side thinks

The clean version of the agent architecture puts all the judgment in one place. The model reasons and decides. Tools execute, deterministically, and return facts. That division is what makes the system legible: there is one reasoner, its decisions are visible in one trace, and everything else is a function call.

Real tools do not stay inside that line. The moment a tool’s honest output is larger or messier than what the caller can use, the tool has to make a judgment about what to return, and no amount of schema design turns that judgment into a lookup. You can push the work back to the orchestrating model by returning the raw payload, and sometimes that is exactly right. It keeps one reasoner in the system and keeps its reasoning visible. But it also means the fifty-page contract lands in the orchestrator’s context in full, that the orchestrator now spends its attention and its budget on extraction rather than on the plan it was actually pursuing, and that a run doing this across a dozen tool calls will exhaust its context long before it exhausts its task.

So there is a genuine fork, and it is worth naming plainly because it is the decision underneath everything that follows. Either the caller pays context to do the interpretation itself, or something closer to the data does the interpretation and returns a smaller, more useful answer. The second option is frequently correct. What it requires is a second inference call, made on behalf of a component that is not the orchestrator. The whole question is where that call comes from.

A second API key solves the immediate problem and creates four durable ones

Handing the component its own model credentials makes the problem disappear in an afternoon, which is precisely why it is so common. The costs arrive later and separately, which is why they are so rarely attributed back to the decision.

Keys accumulate. Each integration that holds a key is a key that has to be provisioned, rotated, scoped, and revoked, and integrations are exactly the parts of a system that get deployed by different teams on different schedules. The set of things holding model credentials becomes the set of things you cannot fully enumerate.

Spend goes dark. Inference initiated inside a tool is real money spent inside your run, and it appears on no dashboard that belongs to the run. This is not a reporting inconvenience. If your system enforces a per-run budget, and a tool the run invoked spends independently against its own account, then you do not have a ceiling. You have a count that omits an unknown share of the spend while presenting itself as complete, and a number like that does active harm, because it tells you that you are safe.

Data egress fragments. Your rule about which records may be sent to a model is only as strong as the number of places it is enforced. When one component holds a key, the rule lives in one place. When six do, it lives in six, five of which will drift, and the answer to “which model saw this customer’s data” stops having a single owner.

Model policy dissolves. The choice of which model handles which class of work, which is one of the few levers that meaningfully moves cost and latency, is now made independently inside components you do not control and cannot change without a deploy in someone else’s repository.

A shared library is a convention, and a delegated call is a chokepoint

The natural objection is that none of this requires inverting the call. Ship every component a common inference client, configured from central config, and you get one place to rotate credentials, one place to set the model, and one place to log. That is a real improvement over scattered keys, and if it is all you can get, take it.

It is still a different kind of thing, and the difference is the whole argument. A shared library standardizes how a component makes a call. It does not move the decision. The component still decides, in its own process, that a call happens, what goes into it, and what comes back. Your control exists only to the degree that every component keeps using the library the way you intended, which is a property of everyone’s discipline rather than a property of the architecture. Conventions bend under deadline pressure, and nothing announces when they do.

Delegated inference moves the decision across the boundary. The component can only ask. The runtime is the thing that answers, which means the runtime can refuse, substitute a cheaper model, strip a field it will not let leave the building, charge the call against this run’s budget, record it, or hold it until an operator looks at it. None of that is available to a library sitting inside the caller’s process, because a library cannot decline on behalf of a policy it is a guest of. A convention is what a system does while everyone is paying attention. A chokepoint is what it does regardless.

The split of control is the entire design

Delegation only works if the two sides own different things, and the boundary is sharper than it first appears.

The requesting component owns the content and the intent. It knows its own domain, it knows what it just fetched, and it alone knows what it is trying to find out. It assembles the request, chooses how much of what it fetched belongs in that request, and states the shape of answer it can actually use. That authority is not negotiable, because the runtime has no idea what a contract-summarizing tool needs summarized.

The runtime owns the execution. It holds the credentials, decides what may be sent and where it may go, charges the cost to the run that caused it, writes the log, picks the model, and decides whether a human sees this first. It answers for all of it, so it decides all of it.

Between them sits a narrow, deliberately weak channel: the component may express preferences. It can say that this task needs a capable model rather than a fast one, that latency matters, that a cheap model will do. Those preferences bind nothing. The runtime can take the hint, route the work somewhere cheaper, or disregard it outright, and none of those is a malfunction. This is easy to read as a courtesy and it is not one. A binding preference would hand the two things the runtime is accountable for, cost and data policy, to a component the runtime did not write and may not trust. The moment a component can compel a model choice, it can compel an expensive one, and the moment it can compel a destination, your egress rule is advisory instead. Advisory hints keep the requester expressive and the runtime sovereign, which is the only stable arrangement when the two are owned by different people.

Centralizing model access is a control win, not a trust win

There is a misreading of all this that is easy to fall into, and it is worth killing early. Routing every inference through the runtime does not make the inference safe. It makes it governed. Those are different properties and only one of them is delivered.

Consider what a component’s inference request is usually built from. The tool fetched something: a document, a support ticket, a record, a web page. It now wants a model to interpret that thing, so the fetched content goes into the request. That content is untrusted by construction. Nobody on your team wrote it, and if it contains text engineered to steer a model, the runtime is about to feed that text to a model holding the runtime’s credentials, and then hand the output back to a component that will act on it, or to an orchestrator that will treat it as a finding.

The centralization has bought you a great deal here, and it is important to be precise about what. It gives you one place where that content is inspected, one place where the request can be refused, one place where the resulting call is logged, and one identity boundary that the injected text cannot cross by itself. That is exactly the value. What it does not give you is any assurance that the model’s answer is uninfluenced by the content that provoked it. Treat the result of a delegated inference the way you treat any other model output derived from untrusted input: as an assertion that still has to earn its place, constrained by whatever validation you would have applied anyway, never as a fact sanctified by passing through your control plane.

Hidden inference is hidden failure

An underrated consequence of letting components think is that the system’s output stops distinguishing between what was retrieved and what was generated.

The orchestrator asks a tool for the terms of a contract. The tool fetches the document, runs an inference to extract the terms, and returns a tidy structure. That structure reaches the orchestrator through the same channel, in the same shape, carrying the same implicit claim to be a fact as a database read. But one of those is a fact and the other is a model’s rendering of a fact, and only one of them can be quietly, confidently wrong in a way that survives every downstream check.

This is the cost of the pattern that nobody mentions, and it compounds. An orchestrator reasoning over a mixture of retrieved facts and generated interpretations, with nothing in either to say which is which, cannot calibrate its own confidence, cannot decide which claims deserve verification, and cannot tell a user which parts of an answer rest on the record and which rest on a summary of the record.

The obligation that follows is small and almost always skipped: when a component’s response contains anything a model produced, the response should say so, and it should be possible to reach the underlying material. Not a paragraph of caveats, just an honest structural distinction between the parts of the result that were read and the parts that were inferred. Provenance costs nothing to attach while the result is being produced and cannot be recovered once the result is in someone else’s hands, and a run that loses track of which of its inputs were generated has lost the ability to reason about its own reliability.

A component may use a model to shape its answer, never to choose its actions

Once tools can think, the architecture has a hole in it, and without a rule the hole widens until nothing is legible. Every integration becomes a small agent, every tool call hides an unknown amount of reasoning, and the system’s behavior stops being reconstructable from the orchestrator’s trace. The rule worth holding, and defending, is this: a component may use inference to shape its own output, and it may not use inference to decide what it does.

Shaping is compressing a payload, extracting fields from a blob, classifying a record, ranking results, translating a phrase into a query dialect. In every one of those, the component’s effect on the world is already determined. The inference changes what comes back, not what happened. That is a bounded, single-call use of judgment interior to a job the orchestrator already authorized, and it is exactly the case delegated inference was built for.

Deciding is different in kind. The moment a model call inside a component determines which record gets written, which downstream service gets invoked, or whether an irreversible action fires, the component has taken the orchestrator’s job. It is now planning, with its own reasoning, under authority it was never granted, and its choices do not appear in the trace anyone will read during the incident. That thing is an agent. If you want it, model it as one: give it an explicit budget, an explicit and narrow authority, a boundary you can reason about, and a place in the trace. What you must not do is let it exist by accident, with nothing at the boundary to announce that it is there.

The same rule bounds the recursion problem, which is otherwise unbounded by construction. A model calls a tool, the tool requests inference, that inference could in principle call a tool, and so on. The single-call shaping discipline keeps the depth at one and keeps the cost of any tool call estimable. If you allow more than that, then you have built a recursive system and you owe it what any recursive system requires: a hard depth limit and a fan-out limit, both enforced where the recursion is actually visible, which is the runtime and not any component inside it. No component can see the depth it sits at. That alone settles where the counter lives.

An approval gate that fires on everything certifies nothing

Because delegated inference passes through the runtime, a human can be placed in front of it, and this is genuinely useful. An operator can see exactly what a component wants to send to a model before it goes, and can refuse. For a small number of consequential cases, that is a real control and sometimes a compliance requirement, since it produces a durable record naming the person who approved a particular disclosure.

It fails in the obvious way. A gate on every inference request, in a system doing real volume, produces more review than anyone can perform, and what cannot be performed gets waved through. The approvals keep accumulating, and they now attest to nothing while looking exactly like they attest to something, which is the dangerous state, because the record will be trusted later by someone who was not in the room.

Gates have to be selective, and the selection criterion is consequence, not caution. Route to a human the requests that send data across a boundary that matters, that spend beyond a threshold, that touch a class of record you have decided is sensitive. Let the rest through, logged. What you gate is a statement of what you actually consider dangerous. Gating everything states nothing.

Where holding the credentials is the right answer

None of this argues that every model call in a system must pass through one runtime, and the exceptions are cleaner than they look.

The first is the one people conflate with this pattern most often. A third-party service that happens to use a model internally is not a component of your agent. It is a service. Its inference is its own: its bill, its data-handling terms, its model choice, part of the product you are buying. Delegated inference is the wrong instinct there, and trying to reach inside its boundary is a category error. The distinguishing question is simple. Is this call your inference, meaning your data, your bill, and your policy, or is it theirs? Delegation applies only to the first, and it applies to it completely.

The second is latency. A delegated call is a round trip back through the runtime before it ever reaches a model. On a path with a hard interactive budget, that hop can be the thing that breaks it, and a component that owns its model client and lives next to its data will win. This is a real tradeoff and it should be made explicitly and narrowly, for that path, with the credential scoped as tightly as the exception requires. It is not a license to hand keys out generally.

The third is trust topology. Sometimes the component runs somewhere the runtime cannot mediate for it, in another organization’s infrastructure or behind a boundary that only permits outbound calls. Then the choice is not available, and the honest response is to stop treating that component as part of your trusted interior and start treating it as an external service with its own budget and its own policy, which is the first case again.

The question to ask of every component is whose inference this is

The durable rule underneath all of this is short. Model access is a privileged capability, it belongs to the layer that answers for cost, data, and policy, and any component that needs judgment should request inference rather than hold the means to produce it.

That rule survives the protocol it is currently expressed in, because it is not really about protocols. It is about where accountability sits. When a component holds a key, the system has quietly distributed its most expensive, most sensitive, and least observable operation across everything that happens to need it, and it has done so in a way that no dashboard will show and no budget will catch. When the component asks instead, the runtime keeps the one thing it cannot afford to lose: one layer that can see, price, constrain, and refuse any inference the system performs.

So the question worth asking of each component is not whether it needs a model. Many of them legitimately do, and pretending otherwise just pushes the work back into the orchestrator’s context where it does not belong. The question is whose call it is. If your data goes into it, your money pays for it, and your policy governs it, then it is your call, and it should be made where you can see it.