The Fan-In Is Where Parallelism Gets Hard

Dispatching work to run concurrently is the easy half of parallel execution, and it is the half everyone thinks about. A single control point splits into several branches, each branch runs at the same time as the others, and the total time drops from the sum of the branches to the length of the longest one. That part is close to free to write. The engineering, and every failure mode worth worrying about, lives on the other side: the point where the branches have to come back together. The fan-out is a decision. The fan-in is a design. Treating them as equal halves of the same easy pattern is how a parallel pipeline ends up faster and quietly wrong.

The reason to say this plainly is that the latency win is genuine and it is seductive. Five independent tasks that each take ten seconds take fifty seconds in a line and ten seconds side by side, and that arithmetic is correct. But the arithmetic hides a bill. Running branches concurrently introduces a class of problems that a strict line never had: branches can fail independently, they can return in any order and in varying numbers, they can race each other for shared state, and they cost tokens in proportion to their number rather than their depth. All of that comes due at the join. The synchronization point is not a formality where results are collected. It is where the price of concurrency is paid, and paying it correctly is the whole job.

Independence is the precondition, not a nicety

Before a single branch is dispatched, the design rests on one claim: the branches are independent of each other. If that claim is false, parallel execution does not just run slower than hoped. It produces wrong results, and it produces them nondeterministically, which is the worst kind of wrong because it survives testing and fails in production. A fan-out with a hidden dependency between two branches is a race condition with extra steps.

Independence is a specific property, and it is worth testing rather than assuming, because the code that reads as parallelizable and the work that actually is parallelizable are not the same set. Three questions settle it for any pair of branches. Can the second branch start without waiting on anything the first branch produces? If it needs that output, there is a data dependency and the two cannot fan out; they have to be ordered. Do the branches write to any shared state, a record, a file, a counter, that another branch also touches? If they do, the value that survives is set by whichever branch writes last, and a concurrent dispatch fixes no order at all. Would running the branches in every possible order produce the same result? If the answer depends on the order, the branches are coupled through something, even if it is not obvious what, and they are not safe to run together.

The common thread in all three is shared mutable state and hidden ordering. Two branches that only read their own inputs and write their own outputs are trivially independent; they cannot interfere. The moment two branches share a mutable resource, or one branch’s correctness depends on another having already run, the independence is gone, and no amount of concurrency machinery will restore it. When the test reveals a dependency, the fix is not to add locking and press on. It is to recognize that those branches are sequential work that a fan-out cannot change, and to order them, or to remove the dependency at its source so they can genuinely run apart. Parallelism is not a thing you impose on work. It is a property the work either has or does not.

The fan-out is a dispatch; the fan-in is a design

Once independence holds, the fan-out itself is mechanically simple. One point of control triggers several subtasks at once, hands each its own inputs, and lets them run. The branches need not be small; a branch can be a single tool call or an entire delegated agent running its own internal pipeline. What makes the fan-out easy is exactly that it demands nothing of the branches except that they start. It is a broadcast.

The fan-in demands everything the fan-out did not. It has four distinct responsibilities, and each has to be designed on purpose rather than falling out of whatever the code happens to do when the branches return. It collects the outputs of the branches that succeeded. It handles the branches that failed, which is a separate concern from collecting the ones that did not, and the hardest of the four. It reassembles results into the right order when the next stage cares about order, because concurrent branches finish in an order determined by their runtimes, not by their dispatch. And it merges the collected outputs into a single result the pipeline can carry forward. A fan-in that does only the first and last of these, grabbing whatever came back and passing it on, is not a simpler fan-in. It is a broken one that has not encountered its first failure yet.

There is a structural fact about the join that reframes the latency win, and it is the barrier property. The fan-in cannot proceed until every branch it is waiting on has arrived, which means the join runs no faster than the slowest branch. The latency floor of a fan-out is the maximum branch time, not the average, and this is where the tidy arithmetic of five-tens-become-ten meets reality. One straggler sets the pace for the whole group. If four branches finish in a second and the fifth takes twenty, the fan-out took twenty, and the four fast branches spent nineteen seconds waiting at the barrier for the slow one. This is why a wide fan-out over branches with high variance in their completion times often disappoints: the tail dominates. The win is bounded by the slowest branch, while the bill is summed over all of them. That asymmetry, latency capped at the maximum and cost accumulated over the total, is the shape of the whole tradeoff.

Partial failure is the cost sequential execution never charged

A strict sequential pipeline has a clean failure story: when a step fails, the steps behind it have nothing to run on, so the whole thing halts. There is one failure, in one place, and everything after it simply does not happen. Parallel execution gives that up. When several branches run at once, some can succeed while others fail, and the pipeline is left holding a partial result: three of five answers, with two branches that errored. Sequential execution never had to decide what to do in that situation because it could never be in it. Parallel execution is in it on every run where anything can go wrong, which is every real run.

So the fan-in has to carry an explicit policy for partial failure, chosen in advance, and there are only three honest options. It can fail the whole operation on any branch failure, which is correct when every branch is required and a partial answer is not an answer at all. It can proceed with the partial result, continuing on what succeeded and marking what did not, which is correct when the branches are additive and a degraded result still has value. Or it can single out the branches that failed and run just those again, leaving the successful ones untouched instead of collapsing the whole fan-out and starting over, which is correct when the failures look transient and the branches are safe to repeat. Retrying deserves a caveat the other two do not: a branch is only safe to re-run if running it twice is equivalent to running it once, so the retry policy quietly imposes an idempotency requirement on any branch that mutates anything.

What is never an option, under any of the three policies, is the thing an undesigned fan-in does by default: take whatever results are ready, drop the failures on the floor, and continue as though the set were complete. This is the cardinal sin of parallel execution, and it is dangerous precisely because it looks like success. The pipeline returns a result. The result is well-formed. It is simply missing the contribution of the branches that failed, and nothing in the output says so. A silently dropped branch turns a loud, immediate failure into a wrong answer that surfaces far downstream, if it surfaces at all, and by then the connection back to the branch that vanished is long gone. The rule is to make the absence of a branch’s result as loud as its presence. Whatever the policy, the failed branches must be accounted for explicitly, because an unaccounted-for failure is indistinguishable from a success that happened to produce less.

The merge strategy is chosen before the branches, not after

The last responsibility of the fan-in, combining the surviving results into one, is not a mechanical afterthought either, because how the results combine determines what each branch was supposed to produce in the first place. The merge strategy is a contract imposed on the branches, and choosing it late means designing the branches against a contract that does not exist yet.

Three merge shapes cover most cases, and they make different demands. A voting merge runs the same question through every branch and takes whichever answer the majority converge on; it trades extra branches for resilience against any single one being wrong, and it only makes sense when the branches are genuine duplicates whose answers can be compared head to head. A concatenation merge gives each branch its own disjoint portion of the work and stitches the pieces back into one collection; it depends on the partition being exact, no region covered twice and none left out. A structured-aggregation merge has each branch fill one named field of a shared object, with the join assembling the fields into the finished record; it depends on every branch knowing which field is its responsibility and returning it in a shape that sits cleanly alongside the others. These are not interchangeable. A branch designed to cast a vote and a branch designed to fill one field of an object are different programs with different outputs, and which one you are building is decided by the merge, not the branch. Deciding the merge first is what keeps the branches from being redesigned once their results turn out not to fit together.

Latency is bought with tokens, and the market has a supply limit

Parallelism trades one resource for another, and the resource it spends is easy to overlook because it does not show up on the clock. Running three branches at once does the work of all three in the time of the slowest, but it consumes the tokens of all three, and it consumes them in a compressed window. The total token cost of a fan-out is the sum across every branch, not the cost of the longest one, so the latency win and the cost are computed against opposite quantities: time against the maximum, spend against the sum. A wide fan-out can cut wall-clock time to a fraction while multiplying token spend by the number of branches. Whether that trade is worth making depends entirely on which resource is scarce for the system in question, and that has to be modeled deliberately rather than assumed away because the latency number is the one that feels like progress.

There is a second limit that turns the theoretical latency win into a smaller real one, and it is the supply side of concurrency. The arithmetic that says five branches finish in the time of one assumes all five actually run at the same time. Rate limits and concurrency quotas frequently make that false. If the ceiling on simultaneous requests is lower than the width of the fan-out, the branches do not all run at once; they queue, and the fan-out silently degrades toward the sequential timing it was meant to escape. A fan-out of fifty branches against a ceiling of ten is really five sequential waves of ten, and its latency reflects that, not the fifty-wide dispatch on paper. This is why the width of a fan-out has to be planned against the actual concurrency the environment permits, not the concurrency the code expresses. The pattern promises a speedup proportional to the parallelism, and the environment decides how much parallelism you are allowed to have.

Parallelize where the data allows and the join is worth paying for

Parallel execution is the right structure exactly where two conditions hold at once: the branches are genuinely independent, and the synchronization tax is worth the latency it buys. The first condition is about correctness and is non-negotiable; branches that share mutable state or hidden ordering are not candidates for a fan-out at any price, because concurrency will not make coupled work independent, it will only make its failures nondeterministic. The second condition is about economics and is a real judgment; even perfectly independent branches are not worth parallelizing when the token cost, the merge complexity, and the partial-failure handling together outweigh a latency win that a straggler may erase anyway. Independence makes a fan-out possible. It does not make it worthwhile.

When both conditions hold, the discipline is to design the join with the same seriousness as the split, because the split is where the speed comes from and the join is where the correctness does. Run the independence test on every pair of branches and order the ones that fail it. Choose the partial-failure policy before dispatching, and make sure every failed branch is accounted for rather than dropped. Choose the merge strategy first, and let it define what each branch returns. Model the token cost as the sum and the latency as the slowest branch, and size the fan-out against the concurrency the environment actually grants. Do that, and parallelism delivers what it promises: the same work in less time. Skip it, and you get a pipeline that is faster on the happy path and silently incomplete on every other one, which is a worse thing to own than the slow sequential version it replaced.