Your Agent Fails Silently While Every Test Stays Green

A team I was helping ran an agent that reconciled invoices against a payments ledger. One afternoon it told a finance user that three invoices were fully settled and the account was clear. The account was not clear. One payment had never posted, a tool call had returned an empty page, and the model had summed what it did receive, decided the total matched, and written a calm paragraph explaining that everything reconciled.

Nothing threw. The HTTP calls returned 200. The JSON validated. The run was marked success, latency was normal, the cost was normal, and every integration test in CI was green that morning and stayed green all week. The only reason anyone found out was that the finance user happened to know one of those invoices was still open and said so four days later.

This is the failure mode that actually costs you in production, and it is the one almost nothing in a normal stack is built to see. The code did exactly what it was written to do. Only the content was wrong, and content is the one thing your test suite was never asked to judge.

Why silent failures outlive your test suite

A longitudinal study of a single production agent runtime put numbers on this. Over eight weeks the author logged 22 incidents with full postmortems, and the headline finding is uncomfortable: around 70 percent of the silent failures were caught by a human looking at the output, not by any test, audit, or health check. (When Errors Become Narratives, a taxonomy of silent failures in a production LLM agent runtime, June 2026.) A retrospective audit of fifteen of those incidents prevented exactly zero in advance and caught 87 percent of them later as regression blocks. Audits, in other words, work as regression engines, not as predictors.

The reason is structural. Tests assert that the machinery ran. The function returned, the schema validated, the service answered, the status was success. A silent failure leaves all of that intact and corrupts only the content. Worse, the same study found that the longest-lived failures, some surviving up to sixty days, lived at the seams between components, exactly where no single test owns the behaviour. Each piece works in isolation. The join does not, and nobody wrote an assertion for the join.

This is why offline evals are necessary but not sufficient. A living eval set catches the regressions you can reproduce, and it is worth building, which is why eval-set drift and a living eval set fed by production traces matter so much. But an eval set only sees the inputs you thought to put in it. The silent failure in production is, by definition, a case you did not anticipate, running against data you did not have at eval time.

The fail-plausible problem

The dangerous part is not that the agent is wrong. Systems are wrong all the time. The dangerous part is that a language model is a narrative engine, so when a step goes wrong it does not stop: it writes a story that absorbs the error and reads as success. The study above calls this the fail-plausible pattern and names it the most dangerous of its five failure classes.

A function that fails hard gives you a stack trace. A model that fails plausibly gives you a confident paragraph. The empty tool result above did not become an error message; it became "I reviewed the three invoices and all are settled." The model took a missing input and turned it into a reassuring conclusion, because producing fluent, reasonable-sounding text is precisely what it is optimised to do.

This is the same instinct behind an agent that would rather guess than admit it does not know. Abstention is the prevention side of the problem: teach the agent to stop when its confidence is low. Detection is the other side, and you need both, because the agent will not always know it is guessing. The fail-plausible run genuinely believes its story. So your detection cannot ask the model whether it succeeded. It has to look at the evidence underneath the story.

Cheap checks first: deterministic invariants

Before you reach for anything statistical, write down the things that must be true about a correct run and check them in code. These are deterministic invariants, and they are the highest-leverage thing you can add, because they cost microseconds, they never hallucinate, and on a well-chosen invariant they almost never false-alarm.

Three kinds carry most of the weight. Recompute any quantity the agent reports, from the tool results it actually received, and compare. Confirm that the tool calls a correct run requires were in fact made, not skipped. And validate the shape of every tool result before the model is allowed to reason over it, which is the same discipline that keeps almost-valid JSON from silently breaking you.

def check_invariants(run) -> list[str]:
    alarms = []

    # 1. Recompute the reported total from raw tool results.
    claimed = run.final["total_settled"]
    actual = sum(p["amount"] for p in run.tool_results("get_payments"))
    if abs(claimed - actual) > 0.005:
        alarms.append(f"total mismatch: said {claimed}, results sum to {actual}")

    # 2. Coverage: every invoice in scope must have been fetched.
    fetched = {r["invoice_id"] for r in run.tool_results("get_payments")}
    for inv in run.input["invoices"]:
        if inv not in fetched:
            alarms.append(f"invoice {inv} never fetched, cannot be reconciled")

    # 3. Shape: a page the model read must not be empty-but-ok.
    for r in run.tool_calls("get_payments"):
        if r.ok and not r.result.get("payments"):
            alarms.append(f"empty page treated as success: {r.args}")

    return alarms

In the real-time detection work by Sunny Dubey, deterministic checks like these caught 60 percent of failures on their own, and 96 percent once a coverage check was added, at zero false positives across 1,825 healthy episodes. (Real-Time Detection and Repair of LLM Agent Failures, August 2026.) That is a remarkable return for code with no model in it. The coverage check, "every required call was actually made," is the one most teams forget, and it is the one that would have caught the empty invoice page. The reason invariants matter so much is the same reason deterministic replay beats guessing when you debug: a fact you can recompute is worth more than any amount of model opinion about whether the run went well.

When invariants run out: a monitor on healthy traces

Invariants only cover failures you can state as a rule. Plenty of silent failures are drift you cannot write down: the agent takes a slightly odd path, calls a tool in an unusual order, produces a step whose shape is fine but whose trajectory is off. For those you train a monitor, and the key design choice is to train it only on healthy runs.

This is one-class anomaly detection. You never try to enumerate the ways a run can fail, because you cannot. You model what normal looks like from your successful traces, score each step of a live run against that model, and raise an alarm when the run drifts far enough from normal for long enough. The signal is the telemetry you already collect if you have done the work on agent observability and GenAI tracing: per-step tool latency, result sizes, token counts, retry counts, the tool call sequence.

# Train once on HEALTHY runs only. No failure labels needed.
features = [step_features(s) for run in healthy_runs for s in run.steps]
monitor = OneClassMonitor().fit(features)   # e.g. one-class ESN + CUSUM

# Score live, step by step. CUSUM accumulates small deviations
# so a slow drift trips the alarm, not just one spiky step.
def watch(run):
    score = 0.0
    for step in run.steps:
        dev = monitor.deviation(step_features(step))  # 0 = normal
        score = max(0.0, score + dev - SLACK)         # cumulative sum
        if score > THRESHOLD:                          # 5% false-alarm budget
            return Alarm(step=step.index, kind="trajectory drift")
    return None

The CUSUM part matters. A single odd step is noise; what you want to catch is accumulated deviation, a run that is quietly going off the rails step after step. Dubey's one-class echo-state-network ensemble with CUSUM alarms caught 71 percent of failures at a 5 percent false-alarm budget, AUROC 0.872, at about 200 microseconds per step. The monitor's edge over a memoryless baseline grew the longer after onset you measured, which is exactly the property you want: it is best at catching the slow silent drift the invariants missed, not the obvious trip a rule would already have caught.

Why the monitor has to be recalibrated where it runs

Here is the result that will bite you if you skip it. A monitor trained on healthy traces from one model does not transfer to another model, even on the same tasks with the same tools. In Dubey's experiments a monitor calibrated on one model and pointed cold at another collapsed to roughly chance, AUROC 0.527 with a 75 percent false-alarm rate, and recovered to 0.885 only after it was recalibrated on the target model's own healthy runs.

The reason is that "normal" is a property of the specific model, framework, and prompt, not of the task. A different model produces different token distributions, different tool-call rhythms, different latencies, so its healthy baseline sits somewhere else in feature space entirely. Your old monitor now reads normal behaviour as anomalous and alarms on everything.

The operational consequence is a rule: the healthy baseline belongs to the deployment. You recalibrate the monitor per model, per framework, and after any change to the prompt or tool set large enough to move the baseline, which is most of them. This is the same trap as a model upgrade that silently invalidates your eval gate: the thing you calibrated against moved, and the number you trusted now means nothing. Notably, the deterministic invariants transferred to a new model unchanged, which is one more reason to lean on them first. A recomputed total is a recomputed total no matter which model wrote the number.

Wiring detection to an action

A detector that only logs is theatre. The point of catching the failure at runtime is to do something before the wrong answer reaches the user, and the right action depends on which layer fired. An invariant breach is a near-certain wrong answer, so it should block: refuse to return the result, and either repair by re-running the failed segment or hand off to a human. A monitor alarm is a probability, not a certainty, so at a 5 percent false-alarm budget you route it to review or abstention rather than hard-blocking every flagged run.

def gate(run):
    breaches = check_invariants(run)
    if breaches:                       # certain: do not ship this answer
        log_incident(run, breaches, severity="block")
        return repair_or_escalate(run)

    alarm = watch(run)
    if alarm:                          # probable: degrade, do not hard-fail
        log_incident(run, [alarm.kind], severity="review")
        return abstain_with_handoff(run, reason=alarm.kind)

    return run.final                   # clean: ship it

Repair is worth it when you can afford the extra call. In Dubey's work, rolling back and re-running flagged runs lifted task success from 52 to 73 percent at a cost of about one extra model call per run. Every incident you do catch, block or not, should be recorded as a decision you can later trace and attribute, because the step where you caught the failure is rarely the step that caused it. That is the handoff to finding the decisive step with counterfactual replay: detection tells you the run went wrong, attribution tells you where.

The parts that bite

The invariants are only as good as your understanding of a correct run, and that understanding rots. A new tool, a changed field name, a vendor who starts returning an empty list instead of a 404, and an invariant that used to fire now passes everything, silently, which is its own silent failure. Treat invariants as code with tests: assert that each one fires on a known-bad fixture, so a dead check is loud.

The monitor's false-alarm budget is a real cost, not a formality. At 5 percent, one run in twenty of your healthy traffic gets flagged for review. If you route all of those to a human, you have built an alert stream nobody reads inside a week, and the alarm you needed drowns in it. Size the review path to the volume, or raise the threshold and accept lower recall. This is the same discipline as an SLO and error budget that hides the failures that matter when you average everything into one green number.

Recalibration is operational work that is easy to forget precisely because the system keeps running without it. A stale monitor does not crash. It just quietly stops catching anything, or starts alarming on everything, and either way you are back to a human noticing. Put recalibration on the same release gate as your prompt and model changes, so the monitor cannot drift out of sync with the thing it watches.

And none of this replaces the hard output checks you already owe your users. A monitor is a probabilistic net for the failures you did not anticipate; it is not a substitute for the deterministic output guardrails that stop one customer's data reaching another. Safety-critical egress gets a rule, every time, not a score.

The takeaway

The failures that reach your users are rarely the ones that throw. They are the runs that pass every check and tell a clean story about a wrong answer, and your test suite is structurally blind to them because it asserts that the machinery ran, not that the content was true. You close that gap at runtime, in two layers. Deterministic invariants first, because they cost nothing, never hallucinate, transfer across models unchanged, and on a good coverage check catch most of the clear-cut wrong answers at zero false positives. Then a one-class monitor trained on your healthy traces, to catch the drift no rule can express, recalibrated wherever it runs because normal is a property of the deployment and not the task.

Build the invariants this week; they are a day of work and they pay immediately. The monitor is a second project, worth starting once you have the traces to train it on. If your agent reports success on runs you later discover were wrong, and your dashboards stayed green the whole time, the gap is not a weak model, it is the absence of a check that looks at the evidence instead of the story. If that sounds like your system, I am happy to spend thirty minutes walking through where to put the first invariant. No pitch, just get in touch.