A Customer Hit a Bug Your Agent Will Never Reproduce

A user sends you a message. Your agent, the one that manages their calendar, moved a meeting to the wrong day and emailed the wrong person about it. They attached a screenshot. It is unambiguous, the agent clearly did the thing, and it is bad.

So you open your logs. You have good logs. You can see the run, the tool calls, the model's reasoning, the timestamps. You read through it twice. Then you do the thing every engineer does next: you try to make it happen again. You copy the user's prompt, you run it against staging, and the agent does the right thing. You run it again. Right again. You run it ten more times and it is fine every single time. The bug is real, a customer got hurt by it, and you cannot reproduce it, because the run that broke was one specific path through a system that takes a slightly different path every time it executes.

This is the part of agent engineering that nobody warns you about. Traditional software is mostly deterministic, so a bug report plus a stack trace usually gets you to a reproduction. Agents are not. The same input produces a different execution on every run, and the failure you need to debug already happened, in the past, under conditions you no longer have.

Where the nondeterminism actually comes from

It helps to be precise about this, because people wave at "the model is random" and stop there, and then they set temperature to zero and are confused when runs still diverge.

There are four separate sources, and the model is only one of them.

The model call itself is nondeterministic even at temperature zero. Floating point results vary across GPU batches, mixture of experts routing depends on what else is in the batch, and providers load balance across hardware. On a short answer you might never see it. On a long agent turn with a lot of generated tokens, one flipped token early changes everything downstream.

The tools are nondeterministic because they call the real world. A search returns different results today than yesterday. An inventory lookup returns a number that has since changed. An external API was slow, or rate limited, or returned an error that one time and never again. The agent's whole trajectory depends on what came back, and what came back is gone.

The clock is nondeterministic, and this one bites people who think they have handled everything else. If your prompt contains "Today is August 22, 2026" or your logic branches on the current time, the run behaves differently depending on when it ran. Replaying it tomorrow with a new date is not replaying it.

And the plumbing is nondeterministic. Random ids, request ordering when tool calls run in parallel, retries that fire on one run and not another. Each of these can be the difference between the good path and the bad one.

You cannot make any of this deterministic, and you should not try. What you can do is record what each source actually produced during the run, and then, when you want to debug, feed those exact values back in.

The version with great logs and no reproduction

Here is the shape of the agent loop most teams have. It is instrumented, it traces, it is not naive. It just cannot be replayed, because every dependency reaches out and gets a fresh answer.

// agent.ts
async function runAgent(userPrompt: string) {
  const messages = [{ role: "user", content: userPrompt }];

  for (let step = 0; step < MAX_STEPS; step++) {
    // Reaches the live model. Same input, different output next time.
    const res = await model.chat({
      model: "claude-opus-4-8",
      messages,
      tools,
    });

    if (res.stop_reason !== "tool_use") return res.text;

    for (const call of res.tool_calls) {
      // Reaches the live world. Whatever it returns now is not what it
      // returned during the run you are trying to debug.
      const output = await executeTool(call.name, call.args);
      messages.push(toolResult(call.id, output));
    }
  }
}

Every arrow out of this function points at something that will answer differently later. The logs tell you what happened. They do not let you make it happen again, and those are not the same thing. A trace is a photograph of the run. What you want is the ability to press play on it. That distinction is the whole game, and it is the same reason I keep tracing and replay as separate layers: one tells you what occurred, the other reconstructs it.

Record every boundary, then serve the recording

The fix is a record and replay harness. You put a single interception point on each boundary where nondeterminism enters, and it operates in one of two modes. In record mode it calls the real dependency and writes the input and output to an append only tape. In replay mode it does not call anything, it looks up the recorded output and returns it.

The core primitive is small. A tape is an ordered list of events, and a recorder that either appends to it or reads from it.

// tape.ts
type Event = { kind: string; key: string; value: unknown };

class Tape {
  constructor(
    public mode: "record" | "replay",
    public events: Event[] = [],
    private cursor = 0,
  ) {}

  // Wrap any nondeterministic call. In record it runs `live` and stores the
  // result. In replay it returns the stored result and never calls `live`.
  async intercept<T>(kind: string, key: string, live: () => Promise<T>): Promise<T> {
    if (this.mode === "record") {
      const value = await live();
      this.events.push({ kind, key, value });
      return value;
    }

    const event = this.events[this.cursor++];
    if (!event || event.kind !== kind || event.key !== key) {
      // The replay diverged from the recording. That is itself a finding:
      // the code path changed since the tape was made.
      throw new ReplayDivergence(kind, key, event);
    }
    return event.value as T;
  }
}

Now the agent loop does not call the model or the tools directly. It calls them through the tape. The loop code does not change shape, it just routes its side effects through one function.

// agent.ts, made replayable
async function runAgent(userPrompt: string, tape: Tape) {
  const messages = [{ role: "user", content: userPrompt }];

  for (let step = 0; step < MAX_STEPS; step++) {
    const res = await tape.intercept("model", `step:${step}`, () =>
      model.chat({ model: "claude-opus-4-8", messages, tools }),
    );

    if (res.stop_reason !== "tool_use") return res.text;

    for (const call of res.tool_calls) {
      // Key the tool event by the call id so parallel tools match on replay
      // regardless of the order the promises happen to resolve in.
      const output = await tape.intercept("tool", call.id, () =>
        executeTool(call.name, call.args),
      );
      messages.push(toolResult(call.id, output));
    }
  }
}

Record a run by giving it an empty tape in record mode and saving tape.events when it finishes. Replay it by loading those events into a tape in replay mode and running the identical function. In replay it never touches the model or the network. It walks the exact path the recorded run walked, and if your bug was in that path, it happens again, every time, on your machine, in a debugger.

The clock and the dice have to go through the tape too

The two boundaries above catch the model and the tools, which is most of it. They do not catch the clock and the random generator, and those are exactly the sources people forget until a replay mysteriously diverges. If the agent ever calls the current time or generates a random id, replaying on a different day or with a different seed changes behavior, and your reproduction quietly stops being a reproduction.

Route them through the tape as well, so the replay sees the same instant and the same ids the original run saw.

// deterministic-env.ts
function makeEnv(tape: Tape) {
  return {
    // The recorded run's wall clock, replayed exactly. A prompt that embeds
    // "Today is ..." now reads the same date on replay as it did live.
    now: () => tape.intercept("clock", "now", async () => Date.now()),

    // Ids the run generated, captured so the replay reuses them instead of
    // minting new ones that would change every downstream key.
    randomId: () =>
      tape.intercept("rng", "id", async () => crypto.randomUUID()),
  };
}

The rule underneath all of this: the agent's business logic is allowed to be nondeterministic, but it is not allowed to reach nondeterminism directly. Every call to the outside, to the clock, to the random generator, goes through the one function that can either record it or replay it. Get that discipline right and reproduction stops being luck.

The parts people get wrong

Matching replay events by position instead of by identity. The simplest tape matches the Nth recorded event to the Nth call. That breaks the instant anything runs in parallel or in a slightly different order, because promise resolution order is not stable. Key each event by something intrinsic, the tool call id, the step index, a hash of the request, so the right recording is served no matter what order the calls fire in. Positional matching is the single most common reason a replay harness works in the demo and falls apart on a real agent.

Treating a replay divergence as an error to swallow. When the replay asks for an event the tape does not have, or the kinds do not line up, do not paper over it. That divergence means the code path changed since you recorded, and that is often the most valuable signal you have. It tells you the fix you just shipped actually altered the trajectory, or that a prompt template changed underneath you. Surface it loudly. The divergence point is where your investigation starts.

Recording secrets into the tape. The tape captures raw tool inputs and outputs, which means it captures whatever flowed through, including tokens, personal data, and customer content. A tape is a debugging artifact that now holds sensitive data, so it inherits every obligation your logs have. Redact known secret fields at record time, encrypt tapes at rest, and put them on the same retention and access rules as the rest of your sensitive data. This is the same egress discipline I wrote about for what agents are allowed to emit, applied to the debug trail instead of the response.

Confusing this with checkpoint recovery. A durable execution layer also records events, and the resemblance fools people into thinking they already have replay. They are built for opposite directions. Recovery replays recorded state so a crashed run can continue forward with fresh live calls. Debugging replay refuses to make live calls at all, so a finished run can be reproduced exactly. If your "replay" calls the model again, it is a new run wearing the old run's clothes, and it will not reproduce the bug.

Only recording the runs that succeed. You do not know in advance which run will be the one a customer complains about. If you only keep tapes for errored runs, you miss every failure that returned a two hundred and did the wrong thing anyway, which is most agent failures. Record broadly, keep everything that errored or got flagged, sample the rest, and expire on a window. The tape you wish you had is always the one you decided was not worth keeping.

The takeaway

The reason agent bugs feel uniquely miserable is that the normal loop, reproduce then fix, has its first step quietly deleted. You are handed a failure and no way to make it happen again, so you guess, and guessing at a nondeterministic system is a bad way to spend a week. Deterministic replay puts the first step back. Record every place nondeterminism enters, the model, the tools, the clock, the random generator, and you can take any run that misbehaved and play it back exactly, as many times as you need, in a debugger, offline, for free. It is a wrapper and an append only log, not a rewrite, and it converts "I cannot reproduce it" into "let me pull the tape."

If your agent is in production and your honest answer to "can you reproduce that failure the customer reported" is no, that is the gap worth closing before the next one lands. Book a consultation call and we can look at where nondeterminism leaks into your runs and get you to the point where every failure is one you can replay.