Your Eval Gate Passed and the Rollout Still Regressed

The release that passed every check

A team I was helping shipped a prompt change to a support agent on a Tuesday. The change had cleared the offline gate: a few hundred cases, an LLM judge, a pass rate that went up, not down. It merged, it deployed to everyone, and by Thursday the escalation queue was full of customers who had been told, politely and confidently, to do the wrong thing. The agent had not crashed. No span was red. The pass rate on the next nightly run was still green.

What broke was narrow and real. A new class of refund question had started arriving that month, the prompt change happened to make the agent paraphrase a policy it used to quote verbatim, and the paraphrase was wrong in about one in twelve of those cases. The offline set had two refund examples, both the old kind. The gate was measuring a product that no longer matched the traffic, and it could not see the one input that mattered because that input was not in the set.

This is not a rare story. In an August 2026 VB Intelligence survey reported by VentureBeat, 61% of organizations that run pre-deployment evaluations reported at least one case in the prior year where an agent or LLM feature passed internal testing and then caused a customer-facing failure. The same survey found only 29% rely on real-time automated quality checks on live outputs. The gate is doing its job. The job is just not enough on its own.

Why a green gate still ships a regression

An offline gate answers one question: does this version score well on the cases I wrote down. That is a necessary question. The offline gate in CI is the thing that stops the obvious regressions before they merge, and you should keep it. But it measures the set, not the release, and the set is a snapshot. Every prompt edit, tool-schema change and retrieval reindex moves the product a little further from the cases the set was written against. Keeping a living eval set fed by production traces slows that drift, but it never closes the gap, because the set is always describing last week while the release is about to meet this week's traffic.

There is a second gap the gate cannot cross. The gate runs on inputs. Production runs on inputs crossed with everything else: the retrieval index as it is right now, the tool latency this afternoon, the exact distribution of questions that showed up today, and the long tail of phrasings no one thought to write a case for. A change can be an improvement on every case in your set and a regression on the slice of real traffic that your set underweights. You will not find that by grading harder offline. You find it by grading the release, on real traffic, after it ships.

So the shape of the fix is not a bigger offline set. It is a second measurement that runs in production, on the actual outputs the release produces, and that can stop a bad release before it reaches everyone.

Online evaluation is a different measurement

Online evaluation means scoring the outputs your system produces in production, as they happen, rather than scoring a fixed set in CI. Same judges, same rubric where you can share it, but the inputs are live and you never chose them. It does not replace the offline gate. It catches what the gate structurally cannot: the effect of this release on this traffic.

The cheapest version is shadow scoring. You already emit traces, so sample a fraction of live responses, run the same judge you trust offline, and attach the score to the trace as a first-class signal rather than a log line you will never query. OpenTelemetry's GenAI conventions give this a shape: the evaluation result event carries a gen_ai.evaluation.name, a gen_ai.evaluation.score.value, an optional score.label, and the response id it refers to, so an online score lands in the same backend as the traces you already emit and can be grouped by model, prompt version and release.

# Score a sampled live response and emit it as a GenAI evaluation event.
from opentelemetry import trace
import random

def score_online(response, ctx, judge, sample_rate=0.15):
    if random.random() > sample_rate:
        return  # most traffic is unscored; sampling bounds judge cost
    result = judge.grade(ctx.input, response.text)  # same judge as the offline gate
    span = trace.get_current_span()
    span.add_event("gen_ai.evaluation.result", attributes={
        "gen_ai.evaluation.name": "policy_faithfulness",
        "gen_ai.evaluation.score.value": result.score,   # 0..1
        "gen_ai.evaluation.score.label": result.label,   # pass | fail
        "gen_ai.response.id": response.id,
        "release.version": ctx.release_version,          # which build produced this
    })

Use the same calibrated judge you pinned for the offline gate, pinned to the same version. If the judge that greenlit the build is not the judge watching production, you are comparing two instruments and will chase phantom regressions.

What to actually score on live traffic

Not everything can be judged without a ground truth, and online is where that bites hardest. Split what you score into three kinds and treat them differently.

Reference-free quality is what an LLM judge can rate from the input and output alone: faithfulness to retrieved context, whether the answer addressed the question, format and policy conformance. These you can score within seconds of the response, which is what makes them useful as a release signal.

Guardrail metrics are cheap, deterministic checks that must never regress regardless of quality: schema-valid output, no silent failures where the agent returns a confident non-answer, latency and cost per request, refusal rate. These are the kind of signals the Google SRE Workbook's canarying releases chapter calls out as worth watching: few in number, user-facing, attributable to the change.

Outcome metrics are the truth, and they arrive late: did the user escalate, retry, thumbs-down, or churn. You cannot gate a release on a signal that takes three days to resolve, but you record it and use it to check whether your fast judge-based signals were telling the truth. Treat the fast signals as the gate and the slow outcomes as the audit of the gate.

Canary the release against the incumbent

Shadow scores tell you how a release is doing in absolute terms. They do not tell you whether it is better or worse than what it replaced, because traffic itself moves day to day. The canary method answers the comparative question. Route a small, representative slice of traffic to the new version, keep the rest on the incumbent, and score both populations with the same judge over the same window.

# Split traffic and tag each request with its population for later comparison.
def route(request, canary_fraction=0.05):
    bucket = stable_hash(request.user_id) % 1000
    if bucket < canary_fraction * 1000:
        version, population = CANARY, "canary"
    else:
        version, population = INCUMBENT, "control"
    resp = version.handle(request)
    resp.population = population        # carried into the eval event
    resp.release_version = version.id
    return resp

Two rules from the SRE chapter matter more than any statistics. First, break every metric down by population; a service-wide average will hide a canary that is on fire because the canary is a sliver of the total. Second, keep the aggregation window no longer than the canary itself, and compare the canary against a concurrent control rather than against yesterday, because time introduces its own variation between weekday and weekend, morning and night. A before-and-after comparison across a deploy boundary will blame or absolve the release for things that were really just Tuesday.

Route by a stable hash of a user or session key, not per request, so a given user sees one version and the comparison is not polluted by the same person bouncing between builds mid-conversation. This is the same discipline a model swap needs: the offline gate clears the new model, and the canary confirms the clearance on live traffic before the rollout widens.

The decision: promote, hold, or roll back

A canary is only worth running if its result is wired into the release. The decision is a comparison of two score distributions plus a set of hard guardrails, evaluated on a schedule, with three outcomes: widen the rollout, hold at the current percentage, or roll back.

# Decide the canary's fate from online scores. Conservative by default.
def canary_verdict(canary, control, min_n=200):
    # Hard guardrails: any breach rolls back regardless of quality score.
    if canary.schema_error_rate > control.schema_error_rate + 0.01:
        return "rollback"
    if canary.p95_latency_ms > control.p95_latency_ms * 1.25:
        return "rollback"
    if canary.n < min_n or control.n < min_n:
        return "hold"  # not enough signal yet; do not widen on noise
    drop = control.mean_quality - canary.mean_quality
    # One-sided check: only a real drop blocks; ties and gains pass.
    if drop > 0 and drop > 2 * pooled_stderr(canary, control):
        return "rollback"
    return "promote"

The asymmetry is deliberate. You need strong evidence to widen a rollout and weak evidence to pause it, because the cost of holding a good release for another hour is small and the cost of flooding every user with a regression is not. Require a minimum sample before you trust any comparison, so a five-request canary does not promote itself on noise. When a guardrail trips, roll back on the metric alone and do not wait for the quality judge to agree: a 25% latency jump is a regression whether or not the answers are faithful.

Rollback here means routing, not redeploying. The incumbent is still running and serving most of the traffic, so a rollback is a weight change that takes effect in seconds. This is the same instinct as a release gate on an error budget: the gate asks the measured budget, not the room, and the automation acts before a human has finished reading the alert.

The parts that bite

Judge cost and latency are the first wall. You cannot grade every live response with a frontier model without doubling your bill and your p95, which is why the sample rate in the first snippet is there. Sample, use a cheaper judge for the high-volume guardrail checks, and reserve the expensive judge for the canary comparison where the sample is small and the stakes are high.

Judge drift will fake a regression. If you update the judge prompt or model in the same week you canary a release, you cannot tell whether the score moved because the agent got worse or the judge got stricter. Pin the judge for the duration of a canary and version it in the eval event so you can prove, later, which instrument produced which number.

Population leakage breaks the comparison quietly. If canary and control share a cache, a memory store, or a retrieval index that the canary writes to, the two populations stop being independent and the control starts absorbing the canary's behavior. Either isolate the shared state per population or accept that your comparison has a ceiling, and lean harder on absolute guardrails when you cannot isolate.

Novelty reads as regression. The week a new intent arrives, both versions may score worse on it, and that is the world changing, not your release. This is exactly where online eval and a living eval set do different jobs: the canary tells you the release is not the cause because the incumbent dropped too, and the living set is where that new intent becomes a permanent offline case so the next release is graded on it before it ships.

Content capture has a privacy cost. Scoring live outputs means a judge, and often a trace backend, sees real user content. Redact before you score, keep the retention short, and make sure the online eval path honors the same data boundaries as the rest of the system, not a quiet exception carved out for the sake of a dashboard.

The takeaway

An offline gate and an online eval are not competing answers to the same question. The gate asks whether a version is good enough to ship against the cases you know. Online eval asks whether the version you just shipped is actually better than the one it replaced, on the traffic you did not get to choose. The first is cheap, repeatable and blind to everything outside its set. The second is the only one that sees the release meet reality, and it is the layer most teams are still missing: a clear majority run pre-deployment evals, and a small minority score live outputs in real time. The regression that passed your gate is living in that gap.

If you build nothing else this quarter, build the canary with a hard guardrail and an automatic rollback, and emit one online score next to the traces you already collect. It is a weekend of work, and it changes the worst day of a release from a Thursday escalation queue into a five-minute auto-rollback that most of your users never notice. If your evals stay green while production keeps surprising you, that gap is usually where the surprise is coming from, and it is worth thirty minutes to map it together.