One Flaky Step Is Sinking Your Agent. Vote It Out.

You did the honest thing and measured your agent instead of trusting the demo. You ran each eval case a dozen times, looked at per-step reliability, and one step lit up red. It's the classification that decides whether a support ticket gets auto-refunded or sent to a human. On the same input, run it now and it says refund, run it again and it says escalate. Roughly nine times out of ten it's right, which sounds fine until you remember that the tenth time it wires money to the wrong place.

So you're staring at a 90%-reliable step in a workflow that needs three or four nines. The usual two answers both hurt. Fine-tuning turns a one-line problem into a labeling-and-training project with its own maintenance tail. Routing that call to the biggest frontier model on every request might buy you a couple of points and it definitely triples the bill on your highest-volume path. There's a third option that a surprising number of teams never reach for, and the reason they skip it is that it feels too dumb to work: run the flaky step several times and let the answers vote.

It's not dumb. The error math is genuinely on your side. But it comes with one failure mode that will quietly hand you a wrong answer with total confidence, and if you don't design around that from the start you'll ship something that looks more reliable and isn't.

Why voting works, in one line of arithmetic

Say a step is wrong with probability p on any single independent run. Run it n times and take the majority answer. For the majority to be wrong, more than half of the individual runs have to be wrong at the same time. When the runs fail independently, that's a binomial tail, and tails shrink fast.

Put numbers on it. At p = 0.10, a single run is wrong 10% of the time. Majority vote over 5 runs is wrong about 0.86% of the time. Over 9 runs, about 0.09%. If the step is already better, say p = 0.05, then 5 runs gets you to roughly 0.12% and 9 runs to well under a hundredth of a percent. You spent 5x the compute on that one step and bought back more than an order of magnitude of reliability. That's the whole pitch, and it's why the recent "consensus-driven execution" work can claim four and five nines out of models that are individually mediocre.

Here's the calculator so you're choosing a sample count on purpose instead of picking 3 because it feels tidy:

from math import comb

def majority_vote_error(p: float, n: int) -> float:
    """Error rate of an n-way majority vote when each sample is
    independently wrong with probability p. n should be odd to avoid ties."""
    need = n // 2 + 1  # number of wrong samples that flips the majority
    return sum(comb(n, k) * p**k * (1 - p) ** (n - k) for k in range(need, n + 1))

def samples_for_target(p: float, target: float, max_n: int = 25) -> int:
    """Smallest odd n whose majority-vote error is at or below target."""
    for n in range(1, max_n + 1, 2):
        if majority_vote_error(p, n) <= target:
            return n
    raise ValueError("target not reachable within max_n; p is too high")

# A step measured at 10% error, and we need three nines on it:
print(samples_for_target(p=0.10, target=0.001))  # -> 9
print(f"{majority_vote_error(0.10, 9):.4%}")      # -> 0.0891%

Two things to notice. Keep n odd so a vote can't tie. And measure p for real before you trust any of this, because the whole curve is driven by that one number. A step you assume is at 10% but is actually at 25% will need far more samples than you budgeted, and the calculator will tell you that honestly instead of letting you find out in production. Measuring p per step is exactly the per-step consistency work I walked through in measuring pass^k instead of pass@1; consensus sampling is what you reach for once that measurement points at a specific weak step.

Building the sampler

The mechanics are straightforward. Fan out N calls at a nonzero temperature so the samples actually differ, normalize each answer into a comparable form, and count. The part people get wrong is the normalization. If your step returns structured output and you vote on the raw strings, {"action": "refund"} and {"action":"refund"} count as different answers, and your consensus dissolves into noise for no reason. Canonicalize before you tally.

import asyncio
import json
from collections import Counter
from dataclasses import dataclass

@dataclass
class Consensus:
    answer: str          # canonical winning answer
    agreement: float     # fraction of samples that backed it
    n: int               # samples actually drawn

def canonical(raw: str) -> str:
    """Collapse trivial formatting differences so equal answers bucket together.
    Adapt this to your output shape; for structured output, parse then re-dump."""
    try:
        obj = json.loads(raw)
        return json.dumps(obj, sort_keys=True, separators=(",", ":"))
    except json.JSONDecodeError:
        return raw.strip().lower()

async def consensus_sample(call_step, prompt: str, n: int = 5) -> Consensus:
    """Run the step n times in parallel and return the majority answer."""
    raw = await asyncio.gather(*(call_step(prompt) for _ in range(n)))
    votes = Counter(canonical(r) for r in raw)
    answer, count = votes.most_common(1)[0]
    return Consensus(answer=answer, agreement=count / n, n=n)

That's a working consensus wrapper. call_step is your existing single call to the flaky step, temperature bumped to something like 0.7 so the samples aren't clones. Everything else is counting.

Don't pay for samples you don't need

Running the full N on every call is the version of this that gives consensus sampling its expensive reputation. You don't have to. Most inputs aren't ambiguous, and for those the first two or three samples already agree unanimously. There's no information left to buy by drawing more. So sample incrementally and stop the moment a decision is mathematically locked in, meaning the current leader has more votes than all remaining samples could possibly overturn.

async def consensus_early_stop(call_step, prompt: str, max_n: int = 9) -> Consensus:
    """Draw samples one at a time, stop as soon as the lead is unbeatable."""
    votes: Counter[str] = Counter()
    for drawn in range(1, max_n + 1):
        votes[canonical(await call_step(prompt))] += 1
        leader, lead_count = votes.most_common(1)[0]
        remaining = max_n - drawn
        # If no one can catch the leader even by sweeping every remaining vote,
        # the outcome is settled. Stop here.
        if lead_count - (drawn - lead_count) > remaining:
            break
    return Consensus(answer=leader, agreement=lead_count / drawn, n=drawn)

On clear inputs this returns after two or three calls. Only the genuinely contested inputs fan out to the full budget, which is exactly where you want to be spending. Draw the samples from a cheaper model too, and the arithmetic often comes out below what you'd pay to send every one of these calls to the frontier model, while landing at a lower error rate than that model gives you alone. That is the counterintuitive result from the enterprise consensus work: cheaper models plus voting can beat a single expensive model on both axes at once.

The trap: wrong answers agree too

Now the part that separates a real implementation from a demo. Everything above assumed the samples fail independently. That assumption is doing enormous work, and it is frequently false.

A fresh line of 2026 analysis, sometimes filed under "wrong-consensus agreement," makes the point sharply: repeated samples can agree just as strongly on a wrong answer as on a right one. If the input has an ambiguous phrasing that pushes every sample the same wrong way, or a field is genuinely missing so every sample hallucinates the same plausible default, or the model carries a bias that fires identically each time, then your five samples don't fail independently. They fail together. Voting doesn't correct a shared mistake, it launders it into a confident-looking majority. From the outside, 5-of-5 agreement on a correlated error is indistinguishable from 5-of-5 agreement on a correct answer. The agreement number lies to you.

Which means the one thing you must not do is treat high agreement, by itself, as a confidence score you can wire straight into an auto-approve. Two guardrails keep you honest.

First, force the samples to be able to disagree. Nonzero temperature is the floor, not the ceiling. Vary the framing across samples, reorder the options, or draw from two or three different models rather than the same one N times. Diverse samplers break the shared-failure correlation that consensus quietly depends on. If every path into the step is identical, you are not measuring reliability, you are measuring one opinion five times.

Second, and this is the load-bearing rule: split the outcome into three buckets, not two.

async def decide(call_step, prompt: str, *, strong_threshold=0.8, review_threshold=0.6):
    c = await consensus_early_stop(call_step, prompt, max_n=9)
    if c.agreement >= strong_threshold:
        return {"action": "auto", "answer": c.answer, "agreement": c.agreement}
    if c.agreement >= review_threshold:
        # Split enough to distrust. Break the tie with an independent check:
        # a stronger model, a rules engine, or a second decomposition, not
        # another vote from the same weak sampler.
        return {"action": "verify", "answer": c.answer, "agreement": c.agreement}
    # Real disagreement. The samples are telling you this input is hard.
    return {"action": "escalate", "answer": None, "agreement": c.agreement}

The escalate bucket is the whole point of measuring agreement. Low agreement is not a nuisance to be thresholded away, it's the model raising its hand to say this particular input is genuinely ambiguous. That's the input you route to a human or a stronger verifier, and it's the same triage logic behind a good human-in-the-loop approval gate. The tie-break in the middle bucket has to be something structurally different from the vote, a stronger model or a rules check, because asking the same weak sampler to break its own tie just draws more correlated samples.

Where this earns its keep, and where it doesn't

Consensus sampling shines on discrete, verifiable, high-stakes steps. Classifications, routing decisions, extracted fields, tool-argument selection, yes-or-no gates before an irreversible action. Anything where there's a countable right answer and the cost of being wrong is real. On those steps, spending 3x to 9x the compute to turn one mediocre step into a near-certain one is an obvious trade, especially when the alternative is a refund going to the wrong account.

It does not fit everywhere. Long free-form generation has no clean answer to vote on, since two good summaries won't be string-equal and there's nothing to tally. Steps that are already cheap and reliable don't need it, and wrapping them in a five-way vote just burns money and latency for reliability you already had. And it will actively mislead you on any step whose failures are correlated by construction, because there the majority is just the shared bias with more confidence. Apply it surgically to the one or two steps your measurements flagged, not as a blanket wrapper around every model call in the graph.

The takeaway

When a single step is flaky and it matters, you have more than the two expensive options everyone reaches for first. Sampling that step a few times and voting gives you exponential reliability gains against independent errors, for a linear and often cheaper cost, provided you do three things: measure the real per-sample error before choosing a sample count, force the samples to be able to fail independently, and treat disagreement as a signal to escalate rather than a number to threshold away. Skip that last part and you've built a machine that states its wrong answers more confidently than before.

If you've measured your agent and found a step that's quietly costing you on the tail, that's usually fixable without a training pipeline or a budget blowout. Book a consultation call and we can figure out which of your steps deserve a vote and which just need to be left alone.