Your Agent Reports Success It Never Delivered

A deploy that went green and a database that was empty

A team I was helping ran a nightly agent that reconciled their billing records. It pulled the day's charges, matched them against the ledger, wrote corrections, and posted a summary to a channel. For three weeks the summary said the same reassuring thing: reconciled 1,200 records, 4 corrections applied, no errors. Everyone read the message, saw the green check, and moved on.

Then finance asked why the corrections were not showing up. The agent had been writing them to a table that a migration had renamed a month earlier. Every write returned a result object the agent happily summarized as applied. The underlying call had been failing a soft validation and returning a body that looked enough like success that the model described it as success. The run was green because nothing in the loop had ever gone and looked at the ledger afterward. The agent reported an outcome it had never observed, and we believed the report because it was confident and it was the only thing we could see.

This is not a rare edge. A public tracker of agents breaking production systems, maintained by the team at Eon's incident log, calls out a whole category for it: the agent got its own story wrong. Across their cases, a recurring pattern is agents misreporting what they did, sometimes reporting work that failed as done, sometimes describing side effects that never occurred. The model is a narrator, and the narration is not the outcome.

Why the report is not the outcome

A language model produces the most plausible next tokens given its context. After a tool call, the most plausible continuation is almost always a clean summary of success, because that is what most completions in its training look like. The model does not see the row land in the table. It does not read the HTTP status. It sees a tool result that was handed back into its context, and even that it treats as text to be summarized, not as ground truth to be checked.

Three things make the gap dangerous. First, tool results are often ambiguous. A 200 with an error in the body, a write that silently no-ops because a filter matched nothing, a partial batch where nine of ten succeeded: all of these can render as success in a summary. Second, timeouts break the link between request and knowledge. When a write times out, the operation may have completed on the server while the response was lost. The agent then reports either outcome with the same confidence, because it has no more information than you do. Third, the report becomes input. In a multi-step run, step five reads step four's claim, not step four's effect, so one false summary contaminates everything downstream.

This is close to the problem where agents fail silently while tests stay green, but it is narrower and more tractable. We are not trying to detect every way an agent can go wrong at runtime. We are refusing to mark one specific effectful action done until its effect is proven.

Read back the state you changed

The pattern is one sentence: after any action with a side effect, read the state that action was supposed to change, and compare it to what you expected. Only a matching read advances the run. The agent's message is demoted from evidence to a claim.

Concretely, every effectful tool gets a paired check. A write to a record is followed by a read of that record, asserting the field now holds the new value. A delete is followed by a lookup that must return not found. An external send is followed by a query of the provider for a message with that idempotency key. The check runs against the real system, not against the tool's own return value, because the return value is exactly the thing that lied in the billing case.

The orchestrator, not the model, owns this loop. You do not ask the agent to verify its own work in the same turn, because the same context that produced the optimistic summary will produce an optimistic verification. The verification is code you wrote, keyed to the action's declared intent, and its result is what the orchestrator records. This keeps the control on the right side of a boundary, the same reason you grade the path rather than the final answer when you evaluate a run offline.

What a verified action looks like in code

Wrap the effectful tool so the write and the read are a single unit. The agent calls update_record; it never sees the raw write.

# Every effectful tool returns a VerifiedResult, never a raw API response.
def update_record(record_id: str, field: str, value) -> VerifiedResult:
    # 1. Perform the side effect.
    write = api.patch(record_id, {field: value})          # may time out or soft-fail

    # 2. Read the state back from the source of truth, not the write's body.
    seen = api.get(record_id)                              # independent query

    # 3. Compare the real state to the declared intent.
    if seen.get(field) == value:
        return VerifiedResult(status="verified", observed=seen[field])
    return VerifiedResult(
        status="unverified",
        expected=value,
        observed=seen.get(field),                          # what is actually there
        note="write returned but state does not match",
    )

The agent receives status="unverified" with the real observed value, so its next step reasons about the truth instead of a hopeful summary. The orchestrator refuses to count the action as done. This is the same discipline as treating a tool failure as a first-class signal to classify and recover from, not retry blindly: an unverified write is a failure even when the API said 200.

Verify through the uncertainty that caused the problem

The subtle part is that the read back runs through the same system that just behaved uncertainly, so it inherits that system's consistency rules. If you read before the write has propagated, you get a stale value and flag a success as failed. If the store is eventually consistent, an immediate read proves nothing.

Know your store's guarantees. Amazon S3, for instance, documents strong read-after-write consistency for puts and deletes, and its guidance for concurrent writers is blunt: the best way to determine the final value is to perform a read after both writes have been acknowledged. On a store with that guarantee, an immediate read back is sound. On one without it, you read a primary, or you poll with a bounded backoff until the value settles or a deadline passes.

Timeouts need idempotency, not just a retry. If the write timed out, you cannot safely repeat it without risking a double effect, so you verify first and only then decide. Keying the write makes the whole thing safe: providers like Stripe support idempotency keys for safely retrying requests without performing the same operation twice, so a retry after an uncertain response returns the saved result rather than charging the card again. Pair that with your own idempotency keys on retry-safe side effects and the verify step becomes: look up the effect by its key, and the lookup is the single source of truth.

def send_with_verify(key: str, payload: dict, deadline_s: float = 5.0) -> VerifiedResult:
    provider.send(payload, idempotency_key=key)           # safe to repeat under the key
    start = time.monotonic()
    while time.monotonic() - start < deadline_s:
        found = provider.lookup(idempotency_key=key)      # the effect, by its key
        if found and found.status == "delivered":
            return VerifiedResult(status="verified", observed=found.id)
        time.sleep(0.25)                                  # bounded poll for propagation
    return VerifiedResult(status="unverified", note="no delivered record within deadline")

Make the verification cheap and specific

A verification that checks the wrong thing is worse than none, because it manufactures confidence. The rule is to assert on the effect the action claimed, not on a proxy for it. A 200 is a proxy. A row count is a proxy. The new value in the specific field, the message marked delivered, the object absent after a delete: those are the effect.

Keep the check narrow so it stays cheap. You are not re-reading the whole table, you are reading the one key you touched and asserting one property. That is a single indexed query per effectful action, which is tiny next to the model call that produced the action. Reserve verification for side effects that matter, and skip it entirely for pure reads.

Action	Weak check (a proxy)	Verification (the effect)
Update a field	HTTP 200 returned	re-read the record, the field equals the new value
Delete an object	delete call did not throw	lookup returns not found
Send a notice	provider accepted the request	provider shows a delivered message for the key
Apply N corrections	summary says N applied	count of rows with the new marker equals N

Record the verified outcome where you can see it. Emit a span per effectful action that carries both the claim and the observed result, following the OpenTelemetry GenAI semantic conventions for tool calls, so a trace shows not just that the tool ran but that its effect was confirmed. Now your dashboards count verified actions, and the gap between reported and verified becomes a metric you can alert on instead of a surprise finance finds for you.

The parts that bite

Verifying too soon is the first trap. On an eventually consistent store, an immediate read back will flag real successes as failures and send the agent into needless, sometimes harmful, retries. Match the read to the store's consistency model, poll with a deadline when you must, and prefer looking up a keyed effect over racing a propagation window.

The verification itself can fail or lie. If the read path is down, you get unverified and must decide whether that means roll back, escalate, or hold. Do not let an unverifiable action quietly count as done, and do not let it quietly count as failed and trigger a destructive undo. Unverifiable is its own state, and for risky actions it is exactly where a human belongs, which is the case for approval gates on the actions you cannot safely automate.

There is a cost to getting the assertion wrong in the lax direction. A check that passes on a proxy gives you the same false green you started with, now wearing a verification badge. When an action does fail verification, resist patching the symptom at the failing step; the real cause is often an earlier decision, and you want to find the decisive step with replay rather than add a retry that hides it. And verification is not a license to let agents loose on irreversible actions: reading back a deleted production table confirms, very reliably, that the table is gone.

The takeaway

Agents do not know what they did. They produce a fluent account of what probably happened, and in production that account is right often enough to be trusted and wrong often enough to hurt. The single most useful habit you can add to an agent that touches real systems is to stop believing its reports and start reading its effects. Write, then read back the thing you changed, compare it to the intent, and let only a matching read advance the run. It is one extra query per action and a different default: the report is a claim, the state is the truth.

Start with your riskiest effectful tool, the one whose failures reach customers or money, and wrap it so the write and the confirming read are one unit that returns verified or unverified and nothing in between. If your agents report success more often than your systems can show it, that gap is where the next quiet incident is already forming, and it is worth a short conversation before it finds you.