Your Agent Scores 90% on Evals and Still Fails Customers

You ran the eval suite before the release. It came back at 90%. Everyone nodded, the change went out, and by the next morning support had a queue of people saying the agent did the wrong thing. Not a crash, not an error page, just the wrong thing, confidently. You pull up the failing conversation, run the exact same input through the agent yourself, and it works perfectly. You run it again. Works again. You start to wonder if the user made it up.

They did not. Your eval number lied to you, and it lied in a way that is very easy to miss because the number was not even wrong. The agent really does pass that input about 90% of the time. The problem is that 90% is not a grade, it is a coin with a heavy bias, and you shipped it as if it were a guarantee.

There are two separate lies buried in a single pass@1 score, and once you see them you cannot unsee them. This is the whole reason reliability has become its own line of research this year, with frameworks arguing that pass@1 simply does not describe what a long-horizon agent does in production. Let me show you both lies, then how to measure the thing that actually predicts whether customers trust your product.

The first lie: an average hides how often you flip

pass@1 as most teams compute it runs each case once and marks it pass or fail. Run 200 cases, 180 pass, you report 90%. The trouble is that a language model is not deterministic at any temperature above zero, and even at temperature zero, tool ordering, retrieval ties, and context assembly introduce enough variation that the same input does not always produce the same trajectory.

So that 90% is really an average over a population of runs you only sampled once each. It cannot tell the difference between two very different agents. One agent passes 180 cases every single time and fails 20 cases every single time. Another passes every one of the 200 cases exactly 90% of the time. Both report 90% pass@1. The first is boring and predictable, the kind of thing you can build a product on. The second is a slot machine, and every customer is pulling the lever once.

For anything a customer touches, the slot machine is a disaster even though the average looks identical, because trust is not built on averages. It is built on the absence of surprises. A user who gets a wrong answer once will forgive a tool. A user who cannot predict whether the tool works this time will stop using it. The metric that captures this is pass^k: the probability that the agent succeeds on all k attempts, not just one of them.

The distinction matters because pass@k, the metric a lot of people half-remember from code generation papers, is the opposite. pass@k asks whether at least one of k tries works, which is exactly right when you can generate ten candidate solutions and keep whichever one compiles. Your customer cannot do that. They get one run. So you want pass^k, the pessimist's metric, where every extra sample can only ever lower the score.

Here is how to actually measure it. Run each case k times, and stop reporting the mean as if it were the whole story.

import statistics
from dataclasses import dataclass

@dataclass
class CaseResult:
    case_id: str
    passes: int          # how many of the k runs passed
    k: int               # runs attempted for this case

    @property
    def p_hat(self) -> float:
        # Estimated per-run success probability for this single case.
        return self.passes / self.k

    @property
    def pass_pow_k(self) -> float:
        # Probability ALL k runs pass, estimated empirically: 1.0 only if
        # every run passed. This is the number your customer experiences.
        return 1.0 if self.passes == self.k else 0.0

def run_suite(cases, agent, judge, k: int = 12) -> list[CaseResult]:
    results = []
    for case in cases:
        passes = 0
        for _ in range(k):
            output = agent(case.input)          # real temperature, real tools
            if judge(case, output):             # your existing pass/fail check
                passes += 1
        results.append(CaseResult(case.case_id, passes, k))
    return results

def report(results: list[CaseResult]) -> dict:
    pass_at_1 = statistics.mean(r.p_hat for r in results)   # the comforting lie
    # Fraction of cases the agent got right on EVERY one of its k tries.
    consistently_correct = statistics.mean(r.pass_pow_k for r in results)
    flaky = [r.case_id for r in results if 0 < r.passes < r.k]
    return {
        "pass_at_1": round(pass_at_1, 3),
        "consistently_correct": round(consistently_correct, 3),
        "flaky_case_count": len(flaky),
        "flaky_cases": flaky,
    }

The gap between pass_at_1 and consistently_correct is the size of the first lie, measured directly. I have watched a suite report 0.91 pass@1 and 0.68 consistently correct on the same run. That 23-point gap is not noise. It is a list of cases, named for you in flaky_cases, where the agent is quietly gambling with every request. Those cases are your entire support queue, and pass@1 rendered them invisible.

The second lie: 90% steps do not make a 90% task

The first lie is about variance within a step. The second is about what happens when you chain steps together, and it is the one that turns a respectable per-step number into a genuinely bad product.

Agents are pipelines. Understand the request, pick a tool, fill the arguments, read the result, decide the next move, and eventually answer. Each of those is a place the agent can go wrong. If every step succeeds independently 90% of the time, the task does not succeed 90% of the time. It succeeds at the product of the step probabilities, and multiplication is not kind here.

from functools import reduce

def task_success(step_probs: list[float]) -> float:
    """Probability an independent-step pipeline completes end to end."""
    return reduce(lambda acc, p: acc * p, step_probs, 1.0)

# A uniformly "90% reliable" agent, at increasing task lengths:
for n in (1, 3, 5, 10):
    p = task_success([0.9] * n)
    print(f"{n:2d} steps @ 0.90 each -> {p:.2%} end to end")

#  1 steps @ 0.90 each -> 90.00%
#  3 steps @ 0.90 each -> 72.90%
#  5 steps @ 0.90 each -> 59.05%
# 10 steps @ 0.90 each -> 34.87%

Read those numbers again. The exact same per-step quality that reads as a solid B-plus turns a ten-step workflow into a coin flip that lands wrong two times out of three. Nobody introduced a bug. The agent did not get worse. You just asked it to do more things in a row, and probability did what probability does.

This is why the headline task-completion score is close to useless for improving anything. It tells you the pipeline is bad without telling you where. The fix is to instrument the trajectory and estimate a per-step success rate, so you can see which link in the chain is dragging the product down. The step with the lowest number is where every hour of your effort should go, because in a multiplicative system the weakest step dominates the outcome.

from collections import defaultdict

def per_step_reliability(trajectories) -> dict[str, float]:
    """
    trajectories: iterable of runs, each a list of (step_name, ok: bool).
    Returns the empirical success rate for each named step across all runs.
    """
    hits = defaultdict(int)
    total = defaultdict(int)
    for steps in trajectories:
        for name, ok in steps:
            total[name] += 1
            hits[name] += 1 if ok else 0
    return {name: hits[name] / total[name] for name in total}

# Example output from a real trajectory dump:
#   tool_selection      0.97
#   argument_filling    0.83   <- the leak; fix this before anything else
#   result_grounding    0.95
#   final_answer        0.94
# Product = 0.72, which is your true task success. Argument filling owns the loss.

Argument filling at 0.83 is what is capping the whole task at 0.72. Push that one step to 0.95 and the task jumps to about 0.82 without touching anything else. That is the leverage you cannot see when you only look at the top-line number. It is the same instinct behind treating a nondeterministic agent as something you can replay and inspect step by step, except here you are aggregating across many runs to find the systematic weak link rather than chasing one bad trace.

Do not fool yourself with a small sample

There is a trap waiting on the other side of this. Once you start running each case k times, it is tempting to run each case three or four times, see that it passed every time, and call it consistent. With k that small, "passed every time" is almost meaningless. An agent that truly passes 70% of the time will still run the table on three tries about a third of the time. You will ship it, and you will be surprised.

So report a lower confidence bound on the per-case success rate, not the raw fraction. The Wilson score interval is the standard tool for a proportion estimated from a handful of trials, and its lower bound answers the question you actually care about: given what I saw, how bad could this case plausibly still be.

from math import sqrt

def wilson_lower_bound(passes: int, n: int, z: float = 1.96) -> float:
    """Lower bound of the 95% Wilson interval for a success proportion.
    Use this instead of passes/n so a lucky small sample cannot fool you."""
    if n == 0:
        return 0.0
    phat = passes / n
    denom = 1 + z * z / n
    center = phat + z * z / (2 * n)
    margin = z * sqrt((phat * (1 - phat) + z * z / (4 * n)) / n)
    return (center - margin) / denom

# 3 of 3 passes looks perfect, but the honest floor is not 100%:
print(round(wilson_lower_bound(3, 3), 3))     # 0.439  <- barely better than a coin
print(round(wilson_lower_bound(19, 20), 3))   # 0.751  <- 20 runs earns real confidence

Three out of three has a true floor around 0.44. That is the number to gate on, and it is why a serious release gate runs enough samples to move that floor somewhere you would actually bet on. Spend your sample budget where it changes a decision: cases near the threshold deserve twenty runs, cases that are obviously fine or obviously broken need far fewer.

Wire it into the gate you already have

None of this earns its keep as a dashboard nobody opens. It has to block a bad release the same way a failing test does. If you already gate deploys on evals, and you should, this is a change to what the gate measures, not a new system. It slots in next to the practice of turning every production failure into a CI eval that catches the regression before users do.

def gate(results, min_consistency: float = 0.85, min_case_floor: float = 0.90):
    consistently_correct = sum(r.pass_pow_k for r in results) / len(results)
    weak = [
        r.case_id for r in results
        if wilson_lower_bound(r.passes, r.k) < min_case_floor
    ]
    ok = consistently_correct >= min_consistency and not weak
    if not ok:
        print(f"BLOCKED: consistency {consistently_correct:.2%} "
              f"(min {min_consistency:.0%}); weak cases: {weak}")
    return ok

# In CI:  raise SystemExit(0 if gate(results) else 1)

Two knobs, both meaningful. The suite-level min_consistency is your floor on how often the whole agent behaves. The per-case min_case_floor on the Wilson bound stops a single genuinely flaky case from hiding inside a good average, which is exactly the failure that put people in the support queue in the first place. One caveat worth stating plainly: this whole approach assumes your judge is itself stable, so if you grade with an LLM, its own variance and length bias will leak into these numbers unless you keep it honest, which is a separate discipline I covered in calibrating an LLM judge you can trust.

The takeaway

pass@1 answers a question no customer ever asks, which is "on average, across everything, roughly how often does this work." Your customer asks a much narrower and harder question: "will it work for me, this time, on this task." pass^k and per-step reliability answer that one.

The shift costs you compute, because you are running the suite k times over instead of once. It buys you the ability to tell the difference between an agent that is dependable and an agent that is merely lucky on average, and to see exactly which step is bleeding reliability so you fix the thing that matters instead of guessing. In a market where inconsistency is what makes people quit a product, that is the number worth optimizing.

If you are staring at a strong eval score that does not match what your users are telling you, that gap is measurable, and the measurement usually points at one or two fixable steps. Book a consultation call and we can put a reliability gate around your agent that predicts production behavior instead of flattering it.