An Unannotated Resolution Is a Defect Even When It Is Correct

When a system reads several sources and produces one answer, the sources will sometimes disagree. That is not an edge case to be engineered away. It is the normal condition of any corpus large enough to be worth querying. The design question is not whether disagreement happens but what your pipeline does with it, and there are only two answers: the conflict reaches the consumer of the output, or the system resolves it silently and presents the result as settled.

Most synthesis pipelines take the second path by default, not by decision. Nobody writes the rule “discard the minority source without telling anyone.” It emerges, because a language model asked to produce one coherent answer from contradictory inputs will produce one coherent answer. Fluency is what it is good at. Given two incompatible claims and no instruction about what to do with them, it will pick, blend, or hedge, and the resulting prose carries no trace of the choice. The output is not wrong-looking. That is precisely the problem.

The position worth defending is this: in any system whose output informs a consequential decision, that silence is a defect in its own right, and whether the pick happened to be right does not enter into it. The system made a judgment call that belonged to someone else, and it destroyed the evidence that a call was made.

The failure mode is fluent, which is why it survives review

Fabrication gets caught. A claim with no support anywhere in the retrieved material tends to fail validation, because there is nothing to validate it against. Silent resolution passes every check that fabrication fails. Each claim in the output is grounded. Each one traces to a real passage in a real source. The provenance is intact. The synthesis is, sentence by sentence, defensible.

What is missing is not in the text. It is the second source, the one that said something else, now absent from the output with no marker where it used to be. A reviewer reading the answer cannot see the gap, because the prose closes over it seamlessly. There is no dangling sentence, no unresolved reference, nothing an automated check can key on. Absence is invisible.

This is why the problem cannot be pushed downstream. Any control that operates on the finished prose arrives too late. By the time the text exists, the disagreement it resolved is unrecoverable from the text itself. You would have to re-read every source to discover what the synthesis dropped, which is the work the pipeline existed to save you.

The cost lands as false confidence, and false confidence is worse than acknowledged uncertainty in a way that is easy to underweight. A consumer who is told two sources disagree can investigate, escalate, hedge the decision, or accept the risk knowingly. A consumer handed a confident claim does none of those things, because there is no signal that any of them are warranted. The system has not just made an error. It has suppressed the information that would let anyone catch it.

Disagreement is a finding, not noise in the pipeline

The instinct to treat conflicting sources as a data-quality problem to be cleaned up is the wrong instinct, and it comes from treating retrieval as a lookup. If the corpus were a database, contradictory rows would be corruption. But a corpus is a set of claims made by different parties at different times for different purposes, and its contradictions carry information about the world that no single source contains.

Sources disagree for several distinct reasons, and the reasons demand different handling:

One source is stale. The world changed and one document did not. The disagreement is real, it has a correct resolution, and recency is the axis that resolves it.

The sources are answering different questions. Two figures differ because they measure different populations, apply different definitions, or scope to different jurisdictions. This is the most common false conflict, and it is not a conflict at all. Nothing is wrong. The system aligned two claims that were never about the same thing.

One source is simply wrong. An error, a bad transcription, a number that lost a digit. Neither recency nor authority reliably identifies it.

The parties genuinely dispute the fact. Two competent sources with access to the same evidence reach different conclusions. Nothing in your pipeline can settle this, because it is not settled.

Only the first has a clean automated resolution. The second needs to be caught and dismissed rather than resolved, because “resolving” it invents a contradiction that did not exist and then suppresses one of two compatible facts. The fourth must not be resolved at all; a system that picks a side in a live dispute and reports it as fact is not synthesizing, it is editorializing with the authority of a machine.

A pipeline that collapses all four into “conflict detected, apply rule” is running a rule over cases the rule was never designed for. The taxonomy matters more than the mechanism.

Detection has to precede the prose

The architectural consequence of the above is a constraint on where in the pipeline conflict handling can live. It cannot be a post-processing pass. It has to happen while the claims are still separable, before generation collapses them into a single narrative.

That pushes toward a shape where synthesis is two stages rather than one. The first extracts claims from each source independently, as discrete structured assertions with their origin attached. The second composes those claims into output. Conflict detection sits between them, operating on the structured intermediate, where “source A asserts X” and “source B asserts not-X” are two records that can be compared rather than two clauses competing for the same sentence.

Detection over that intermediate is a two-part test. First, alignment: do these two claims address the same thing? Semantic proximity gets you candidates. Second, contradiction: given that they address the same thing, is one incompatible with the other? Only the conjunction is a conflict. Proximity alone catches every pair of claims about a shared topic, most of which agree or are simply unrelated statements that happen to share vocabulary.

Both tests are themselves probabilistic, and this deserves to be said plainly rather than glossed. You are using inference to decide which claims to flag as contradictory, which means the detector has its own false positives and false negatives, and neither is free. A missed conflict is the original failure, unimproved. A spurious conflict is worse than it looks: it puts a disagreement in front of a reader that does not exist, and a few of those teach the reader to skip the annotations entirely.

The engineering here favors precision over recall in the detector and pushes the recall problem elsewhere, because the annotation channel is the scarce resource, not the compute. A detector that flags everything has the same practical effect as one that flags nothing.

There are two structural limits worth naming. Detection framed as a pairwise comparison scales badly as the source count grows, and more importantly it misses the shape of real disagreement, which is often three sources clustering against one rather than a clean two-way split. Aligning claims into groups and looking at the distribution within a group is a better fit for what actually occurs. And conflicts are frequently partial: two sources agree on the substance of a claim and differ on one qualifier, a date, a magnitude, a scope condition. A detector that treats claims as atomic sees either a total conflict or none, and both readings are wrong.

The resolution rule belongs to the system, not to the model’s judgment

Once a conflict is real, something has to decide. The decision should be an explicit rule the system applies, not a preference the model exercises inside a generation step, and the reason is not that models decide badly. It is that a rule in the system is inspectable, testable, and changeable, while a preference exercised inside generation is none of those. You cannot review a policy you never wrote down, and you cannot fix one that lives in the weights.

Three rules cover most of the ground, and each has a boundary that matters more than its statement.

Prefer the more recent source works where facts supersede: prices, policies, versions, regulations. Its hidden assumption is that time orders the claims monotonically toward truth, and that assumption breaks in ordinary ways. A recent document may be a summary of an older one, inheriting its errors while acquiring a fresher timestamp. A recent document may be a draft, and an older one the ratified version. Most quietly, the timestamp your system has is often not the timestamp the rule needs: ingestion date, last-modified date, and the date the claim was actually asserted are three different things, and pipelines routinely use whichever one the storage layer made convenient. A recency rule running on the wrong date field is worse than no rule, because it resolves confidently in an arbitrary direction.

Prefer the more authoritative source requires a ranking that exists before the conflict does. That is the whole cost of the rule, and it is not a small one. Authority is also domain-scoped rather than global: a source that is definitive about regulation may be secondary about implementation, and a flat ranking over the corpus will confidently apply the wrong hierarchy in the wrong domain. Worse, an authority ranking is a durable institutional judgment encoded in a configuration layer, and it will drift out of date silently, because nothing about a stale ranking looks broken from the outside.

Escalate to a human is the correct answer when neither of the others applies, and it is the rule most often specified and least often honored. Escalation is bounded by a reviewer’s finite attention, so it only works when it is rare, which means the design work is in the criteria that keep it rare.

There is a fourth option that belongs on the list and is usually missing: report the disagreement and resolve nothing. For genuine disputes, this is not a failure to decide. It is the only accurate answer available, and a system that can produce it is more honest than one whose rule set guarantees an answer for every input.

The one strategy to rule out entirely is blending. Averaging two conflicting numbers, or splitting the difference in prose, produces a claim that no source made and no source supports. It has the form of a compromise and the substance of a fabrication.

Resolving a conflict does not close the record

The step that gets dropped, because it delivers nothing at the moment it costs something, is persisting the conflict after it has been resolved. The resolution went into the output. The conflict feels handled. Writing it to a log is work with no immediate consumer.

The consumer is later, and there are three of them.

The first is anyone auditing the output after the fact. Without a record, a resolved conflict and an uncontested fact are textually identical, so a later reviewer has no way to read one as anything but the other. The log is the only thing that distinguishes them, which means it is the only thing that makes the output auditable at all.

The second is the rule itself. A resolution strategy is a hypothesis about how your corpus behaves, and hypotheses are wrong in ways that only show up in aggregate. If the recency rule is reading the wrong date field, no single resolution reveals it. A thousand logged resolutions reveal it immediately, because the pattern of what got discarded stops making sense. The log turns resolution policy from something you assert into something you can evaluate.

The third is the future re-decision. Authority rankings change. A source gets deprecated, a new one becomes definitive, a domain gets recarved. When the ranking changes, every past conflict it resolved is now potentially resolved wrongly, and the only way to find those is to have kept them. A pipeline with a conflict log can re-run its resolutions against a new policy. One without it cannot even enumerate what would need re-running.

The record has to hold enough to reconstruct the decision without the sources: what each party asserted and where it came from, which rule ran, and how it ruled. The claim that lost is the part that gets trimmed for storage and the part that carries the value. A log that records what won documents the answer. A log that records what lost preserves the question.

Conflict rate over that log is also one of the better health signals available for a corpus, and it comes free once the log exists. A rising rate in a domain means the sources are diverging, which usually means something upstream is stale, mis-ingested, or newly contested. That is worth knowing before it reaches an answer.

Annotation is an interface, and its budget is the reader’s attention

Detection and resolution are internal. Annotation is the part the consumer of the output actually experiences, and it is governed by interface economics rather than correctness. An annotation that is technically present and practically ignored has bought nothing.

The scarce resource is attention, and it is spent on every flag whether or not the flag deserved it. Calibration to stakes is therefore not a nicety. A conflict that changes what a reader would do belongs adjacent to the claim it affects, unavoidable, in the path. A conflict that qualifies a claim without changing its use belongs somewhere reachable but not in the way. A conflict that matters only to someone auditing the whole output belongs in a consolidated summary aimed at that reader and nowhere else. Same detection, same log, three placements, chosen by consequence.

Uniform annotation looks like rigor and functions as noise. Flag everything and readers learn the flags do not discriminate, and the ones that mattered are read with the same glance as the ones that did not. This is the same dynamic that turns a monitoring system into wallpaper, and it fails the same way: not with a decision to ignore the signal, but with a gradual, entirely rational discount applied to a channel that has stopped carrying information.

There is a corresponding discipline on the detector. If a class of difference is expected and uninteresting, a figure that varies within tolerance across sources, it should be defined as agreement in the alignment step rather than detected as a conflict and then suppressed in the presentation layer. Tolerances belong in the definition of conflict. Suppression rules in the display layer are a sign the definition was wrong.

The same reasoning applies to conflicts arising between components rather than documents. When an orchestrator merges results from several workers that examined different slices of a problem, their outputs can contradict each other exactly as two sources can, and the merge step will smooth it exactly as fluently. The aggregation point is a synthesis point, and it needs the same treatment.

When not to pay for this

The machinery is not free. It adds a structured intermediate stage, an inference pass, a persistent store, and an interface surface, and it introduces its own false positives. Several conditions make it a bad trade, and they are worth stating rather than implying.

Single-source synthesis has no conflicts to detect, and building the apparatus against a corpus that will never have a second opinion is architecture for its own sake. Low-consequence, high-volume output, where a wrong answer costs a moment and a correction is cheap, does not justify the latency or the reader’s attention. Domains where the sources agree by construction, because they derive from a common authority, will produce a conflict rate near zero, and detection there is a tax that catches ingestion bugs, which is worth something but should be priced as what it is.

The condition that flips the trade is asymmetry of consequence. When acting on a confidently wrong claim costs materially more than pausing on a flagged one, annotation pays, and it pays more the wider that gap gets. That is why the pattern belongs in regulated, clinical, financial, and safety-adjacent synthesis and is overbuilt for a content feed. The question is never whether disagreement exists. It is what a hidden disagreement costs when it turns out to have mattered.

Honest synthesis names what it could not settle

A synthesis pipeline is not a summarizer. It is a system that makes judgments about which claims are true and presents the results as an answer. Those judgments are unavoidable, because the sources force them. What is avoidable is making them invisibly.

The full pattern is four commitments, and they only work together. Detect conflicts while the claims are still separate, because after generation the evidence is gone. Resolve them with explicit rules that live in the system where they can be read, argued with, and changed, and accept that some conflicts have no correct automated resolution and must be reported rather than settled. Log every resolution including the claim it ruled against, because that claim is what makes the decision auditable and the rule improvable. Annotate in proportion to consequence, because attention spent on conflicts that do not matter is not available for the ones that do.

The unifying claim underneath all four is about who is allowed to decide. When sources disagree, someone has to choose, and the only question is whether that someone is a rule you wrote, a reviewer you asked, or a fluent model resolving it in a sentence nobody will ever be able to inspect. A system that reports its disagreements transfers a decision to the person accountable for it. A system that smooths them keeps the decision and hands over the confidence.