Your Agent Dies at Step 40 and Starts Over From Zero

You kick off an agent to work through a sixty-step research task. It plans the questions, searches, reads, drafts. At step forty, a deploy rolls the process, or the model provider throttles you, or a container just dies. When the job comes back, it starts again from step one.

Every token you spent on those forty steps is gone. You pay for them a second time. And if any of those steps did something in the real world, sent an email, wrote a row, kicked off a downstream job, the restart does it all again. Now a customer has two copies of the same message and your database has duplicate records, all because a long run had no memory of what it had already finished.

This is the failure mode that separates a demo agent from a production one. A single LLM call either works or you retry it, and nobody notices. A multi-step run that takes minutes and touches real systems has a much larger surface for something to interrupt it, and the naive answer, start over, is the expensive and dangerous one. The fix is a pattern that backend engineers have used for years and that agent workflows are now rediscovering: durable execution.

The business problem: a run is not a function call

Think about what a long agent run actually costs. Fifty to two hundred model calls, each one billed, plus retrieval, plus tool calls that hit APIs you pay for or that mutate state you care about. When that run restarts from zero, you are not just wasting a few seconds. You are re-spending the entire token budget for everything that already succeeded, and you are re-triggering every side effect along the way.

The retry-the-whole-thing approach makes the second problem worse than the first. Re-spending tokens is money. Re-firing side effects is trust. If step twelve charged a card or notified a user, replaying step twelve from the top does it twice, and no customer forgives that because your infrastructure hiccuped. So the goal is not just to save the tokens. It is to make a resumed run pick up exactly where it left off, without redoing a single thing it already did.

Durable execution: treat the run as a log of finished steps

The core idea is small. Instead of holding the run's progress only in memory, where a crash erases it, you write a durable record after each step completes. The run becomes a log: a list of steps that finished and what each one produced. When the process comes back, it reads that log and skips straight past everything already done.

Start with the store. It needs to survive a crash, so every write has to land atomically.

import json, os

class Checkpoint:
    """A durable log of completed steps for one run, keyed by step id."""
    def __init__(self, run_id, path="runs"):
        os.makedirs(path, exist_ok=True)
        self.file = os.path.join(path, f"{run_id}.json")
        self.done = self._load()

    def _load(self):
        if os.path.exists(self.file):
            with open(self.file) as f:
                return json.load(f)
        return {}

    def record(self, step_id, value):
        self.done[step_id] = value
        # Write to a temp file then rename. os.replace is atomic, so a crash
        # mid-write leaves the old log intact instead of a half-written one.
        tmp = self.file + ".tmp"
        with open(tmp, "w") as f:
            json.dump(self.done, f)
        os.replace(tmp, self.file)

    def has(self, step_id):
        return step_id in self.done

A single JSON file per run is enough to understand the pattern and enough for one worker. In production you swap it for Postgres, Redis, or DynamoDB so multiple workers can resume the same run, but the contract does not change: record after each step, and make the write atomic so the log is never corrupt.

The replay-safety contract

Here is the part people get wrong. You cannot resume by re-running the code and hoping it lands in the same place, because agent runs are full of non-determinism. An LLM call returns something different each time. Retrieval pulls different documents as your index shifts. A timestamp or a random choice changes on every pass. If you replay the code and let those calls run again, the resumed run takes a different path than the original, and you have not recovered the old run, you have started a subtly different one.

The contract is this: every non-deterministic step records its result the first time it runs, and on replay it returns the recorded value instead of executing again. Wrap it once and the rule enforces itself.

def step(cp: Checkpoint, step_id: str, fn, *args, **kwargs):
    """Run fn exactly once for this run. On a resume after a crash, return the
    recorded result instead of calling fn again. Any non-deterministic work,
    an LLM call, a search, a timestamp, must go through here so it is recorded
    the first time and reused on replay, never re-executed."""
    if cp.has(step_id):
        return cp.done[step_id]        # replay: reuse the recorded output
    result = fn(*args, **kwargs)       # first run: actually do the work
    cp.record(step_id, result)
    return result

Now the agent loop reads almost like the version you would write without any of this, except every unit of work goes through step with a stable id.

def run_research(topic, run_id):
    cp = Checkpoint(run_id)
    plan = step(cp, "plan", llm_plan, topic)

    findings = []
    for i, q in enumerate(plan["subquestions"]):
        # Ids are ordered by position, so a resumed run lines up exactly with
        # the first attempt. Never key a step on time or a random uuid.
        docs = step(cp, f"search:{i}", search_web, q)
        answer = step(cp, f"answer:{i}", llm_answer, q, docs)
        findings.append(answer)

    return step(cp, "synthesize", llm_synthesize, topic, findings)

If the process dies at subquestion forty, you restart it by calling run_research with the same run_id. The plan step short-circuits to its recorded value, the first thirty-nine searches and answers short-circuit too, and only the unfinished work actually calls the model. The run resumes at the exact step it died on, having spent nothing to get back there.

The whole thing hinges on the step ids being stable. Key a step on enumerate position and it matches on replay. Key it on datetime.now(), a random uuid, or the iteration order of a set, and the resumed run generates different ids, matches nothing, and re-runs everything, which is the bug you were trying to kill. This is the same discipline I wrote about in context compaction for long-running agents: once a run outlives a single tidy execution, its state has to be something you can reconstruct deterministically, not something you hope is still in memory.

Idempotent side effects close the last gap

Recording outputs handles reads. It does not fully handle writes, and writes are where the real damage happens. Look closely at the ordering: the side effect fires, and only then does the checkpoint get written. If the process dies in that narrow window, after the email is sent but before the record lands, then on replay the step has no checkpoint, so it runs again and sends a second email. The checkpoint alone cannot close this gap, because the durable record is written after the effect, not before.

The answer is to make the side effect itself idempotent with a key the provider deduplicates on.

def send_email_once(cp, step_id, to, body, idem_key):
    if cp.has(step_id):
        return cp.done[step_id]
    # The idempotency key is derived from stable inputs, so a replay sends the
    # SAME key. The provider treats the second send as a no-op and returns the
    # original result, so a crash between "sent" and "recorded" cannot double-send.
    resp = email_api.send(to=to, body=body, idempotency_key=idem_key)
    cp.record(step_id, {"id": resp.id})
    return cp.done[step_id]

# Key derived from run + step, never from time or randomness, so it is
# identical on the first attempt and every replay.
idem_key = f"{run_id}:notify_user"
send_email_once(cp, "notify_user", user_email, draft, idem_key)

With the key in place, the worst case is that the send runs twice, but the second call carries the same key, so the provider drops it and hands back the original result. At-least-once execution plus an idempotent effect gives you effectively exactly-once, which is the strongest guarantee you can get across a boundary you do not control. Almost every payment, email, and queue API supports an idempotency key for exactly this reason. Use it, and derive the key from run and step ids so it never drifts.

This same instinct, do not let a recovery path repeat an action, is what I covered in recovering from tool-call failures. A retry is only safe when the thing you are retrying is safe to run twice, and idempotency is how you make it so.

The tradeoffs worth naming

Durable execution is not free, and treating it as free is how it turns into its own class of bug.

The store is now a dependency on your hot path. Every step does a read and a write, and if that store is slow or down, your agent is slow or down. A file works for one worker; the moment you want multiple workers resuming the same runs, you need a real database and a way to lease a run so two workers do not resume it at once.

Checkpoint size grows with what you record. If every step stashes a full model completion, your log balloons with text, and some of that text is prompt content you may not want sitting at rest. Store references or hashes for large blobs, and keep secrets and sensitive data out of the checkpoint entirely.

Non-determinism you forgot to wrap will bite you quietly. If a step branches on random or the wall clock and you did not route that value through a recorded step, replay takes a different branch than the original run, and the failure is invisible until it produces a wrong answer. The fix is boring and absolute: every non-deterministic value the run depends on is a recorded step.

And do not reach for any of this on short, cheap, side-effect-free work. A three-second call you can just retry does not need a durable log, and the store only adds latency and a dependency. This pattern earns its place on long runs, on expensive token spend, and on anything that writes to the world.

When you outgrow the hand-rolled version, that is the signal to adopt an engine rather than keep extending your JSON file. LangGraph checkpointers and durable-execution runtimes like Temporal, Restate, DBOS, and Inngest give you this exact contract plus multi-worker scheduling, durable timers, and human-in-the-loop signals. The concepts here are the same ones they implement, which is the point: knowing the contract lets you use those tools deliberately instead of cargo-culting them. It pairs naturally with splitting a run across roles, which I covered in moving from a monolithic agent to orchestrator and workers.

The takeaway

A long agent run is not a function call, and treating it like one, where the only recovery is to run it again from the top, is what makes agents expensive and untrustworthy in production. Durable execution changes the unit of recovery from the whole run to the single step. You record what finished, you reuse recorded outputs on replay instead of re-running non-deterministic work, and you guard every side effect with an idempotency key so a crash in the wrong microsecond cannot double-charge or double-send.

Build the small version first. A checkpoint store with atomic writes, stable step ids ordered by position, and idempotency keys on every write. That is a few dozen lines, and it turns a run that loses everything on a hiccup into one that shrugs and picks up where it left off. Adopt a durable-execution engine when you need multiple workers, timers, and signals, not before.

If you are running agents that take real time and touch real systems, and a crash halfway through is currently a disaster instead of a shrug, this is exactly the kind of reliability work I help teams get right. Book a consultation call and we can look at where your longest runs are losing money and trust.