Your Agent Retried and Charged the Card Twice

An agent with read-only tools is forgiving. If a search times out you just run it again, and the worst case is a little wasted latency. The moment you give that same agent a tool that charges a card, sends an email, or creates a shipment, every retry stops being free. A tool call that times out looks like a failure, so your retry logic does the sensible thing and calls it again. Except the first call was not a failure. The remote processed the charge, then the response got lost on the way back. Now the customer has paid twice, and the only record of why is a timeout in a log nobody reads until the refund request comes in.

This is the failure that turns a reliability feature into an incident. Retries are supposed to make an agent more robust, and they do, right up until they land on an operation that already succeeded and quietly ran it a second time. The fix is not to retry less. Timeouts and dropped connections are normal, and an agent that gives up on the first blip is worse, not better. The fix is to make the write safe to repeat, so that calling it twice with the same intent produces the same single effect. That property has a name, and it is idempotency.

Where the duplicate actually comes from

It helps to be precise about the window, because the instinct is to blame the model and the model is not the problem here. A write tool call has three moments: the request leaves your process, the remote commits the work, and the response comes back. A duplicate happens when something breaks between the second and third moments. The work is done. Your client never hears about it. From where your code sits, a committed-but-unacknowledged call and a call that never arrived are indistinguishable, and you have to retry to cover the second case, which means you will sometimes retry the first.

Agents make this worse than a normal service does, for two reasons. First, they retry in more places. There is the HTTP client retry, the tool-execution wrapper retry, and the model itself, which will happily call the tool again if the first result looked like an error. That is three independent layers that can each replay the same write. Second, agents replay. If you checkpoint an agent's state and resume after a crash, the durable execution replay will re-run any tool call that had not been recorded as complete, which is exactly the call that was in flight when the process died. Both paths converge on the same requirement: the write has to be safe to run more than once.

Here is the naive version that ships in most first drafts, and the bug in it.

async def call_tool_with_retry(tool, args, attempts=3):
    for i in range(attempts):
        try:
            return await tool.run(args)
        except (TimeoutError, ConnectionError):
            if i == attempts - 1:
                raise
            await asyncio.sleep(2 ** i)  # backoff, then try again
    # Problem: on a timeout AFTER the remote committed, the next
    # attempt runs the same write again. For charge_card, that is
    # a second charge. The retry is correct; the tool is not safe.

Nothing is wrong with the retry logic. The problem is that charge_card has no way to recognize that it is being asked to do something it already did.

The idempotency key

The mechanism is a stable key that identifies the unit of work, attached to the request, that the executor uses to collapse duplicates. The first time the executor sees a key it runs the operation and records the result against that key. Every later call with the same key skips the work and returns the recorded result. Two attempts, one effect.

The whole thing hinges on deriving the key correctly, and this is where people go wrong. The key must come from the business action, not the HTTP request. If you hash the full request payload including a timestamp, every retry produces a new key and you are back to duplicating. If you generate a random key inside the retry loop, same failure. The key has to be identical across every attempt of the same logical action and different for genuinely different actions. For a charge, that is usually the order id plus the amount. Derive it once, above the retry loop, and pass it down so all three retry layers carry the same key.

import hashlib
import json

def idempotency_key(operation: str, business_id: str, **fields) -> str:
    # Stable across retries: same operation + same business identity
    # + same salient fields => same key. No timestamps, no randomness.
    payload = json.dumps({"op": operation, "id": business_id, **fields},
                         sort_keys=True, separators=(",", ":"))
    return f"{operation}:{hashlib.sha256(payload.encode()).hexdigest()[:32]}"

# Two attempts at the same charge collapse to one key:
key = idempotency_key("charge_card", order_id="ord_8891", amount_cents=4200)
# A different order gets its own key and is never deduped against this one.

Enforcing it with a dedup store

If the remote supports idempotency keys natively, and payment providers generally do, you pass the key through and let the remote enforce it. That is the strongest option because it covers the exact window you cannot see, the gap between commit and acknowledgement, from the remote's side. But plenty of tools call internal services or older APIs with no such support, and for those you enforce it yourself with a small dedup store.

The store needs to handle the case people forget: two attempts running at the same time. A slow first attempt and a retry can overlap, so the store has to reserve the key before doing the work, not after, or both attempts sail through the check and both execute. The pattern is reserve, run, record, with the reservation acting as a lock.

import time

class DedupStore:
    """Backed by Redis or Postgres in production. States per key:
    'in_progress' (reserved, running) or 'done' (result cached)."""

    async def run_once(self, key: str, do_work, ttl_seconds=86400):
        # Atomic reserve: only the first caller wins the insert.
        reserved = await self._claim(key, ttl_seconds)
        if not reserved:
            existing = await self._get(key)
            if existing["status"] == "done":
                return existing["result"]          # duplicate: return first result
            # A concurrent attempt holds the key and is still running.
            # Wait for it to finish rather than starting a second write.
            return await self._await_result(key, ttl_seconds)

        try:
            result = await do_work()               # the real side effect, once
            await self._record(key, result)        # flip 'in_progress' -> 'done'
            return result
        except Exception:
            await self._release(key)               # let a genuine retry try again
            raise

The _claim step is the important one. In Redis it is a SET key value NX EX ttl, which sets the value only if the key does not already exist and returns whether it won. In Postgres it is an INSERT ... ON CONFLICT DO NOTHING and checking the row count. Either way, exactly one caller gets to run the work, and everyone else either gets the cached result or waits for the in-flight attempt to finish. Note the except branch releases the key on a real failure, so that a legitimate retry after a genuine error is not permanently blocked by a stale reservation. Only a completed operation stays locked as done.

Wire the two together and the retry-unsafe tool from earlier becomes safe without touching the retry logic at all.

async def charge_card(order_id, amount_cents, dedup: DedupStore):
    key = idempotency_key("charge_card", order_id, amount_cents=amount_cents)

    async def do_charge():
        # If the provider supports it, forward the SAME key so the
        # remote also dedupes across the commit/ack gap you can't see.
        return await payments.charge(
            order_id, amount_cents, idempotency_key=key
        )

    return await dedup.run_once(key, do_charge)

# Now the retry wrapper can call this three times on a timeout and the
# card is charged exactly once. The wrapper did not have to change.

Tradeoffs and the places it bites

The first thing to get right is what the key covers. Scope it to the side effect, not the inference request. One agent step might be five model calls and three tool calls, and it is only the write tools that need keys, scoped to the business action each one performs. Tag your tools as read or write in their definitions and only wrap the writes. This is the same read/write distinction that your tool authorization layer already cares about, so it is worth having in one place.

The second is the TTL. The dedup record has to live at least as long as the longest possible retry gap, including a crash-and-resume that might come minutes or hours later. Too short a window and a delayed retry misses the record and duplicates anyway. A day is a common default, longer if your durable workflows can resume after long pauses. The cost is storage, which is cheap, so err long.

The third is the failure branch. Be deliberate about which errors release the key and which do not. A network timeout should release, because you genuinely do not know if the work happened and you want the retry to go through the same dedup gate again. A validation error that says the request was malformed should not be retried at all. And the subtle one: if the work committed but recording the result failed, releasing the key lets a retry re-run a committed write. Where that risk is unacceptable, lean on the remote's own idempotency so that the re-run is caught at the source even if your local record is missing. Defense in depth beats trusting one store.

The fourth is that idempotency is not a transaction. It makes a single write safe to repeat. It does not make three writes atomic. If your agent step charges a card, creates an order, and sends a receipt, keying each one individually stops each from duplicating, but a crash between step two and three still leaves you with a charge and an order and no receipt. That is a saga problem, and idempotency is the property that makes each step of the saga safe to retry. It is a foundation, not the whole building. It pairs naturally with the broader tool call reliability work of validating results and recovering from partial failures.

The takeaway

Retries make an agent robust to the network. Idempotency makes retries safe to have. Without it, the more aggressively you retry the more duplicates you create, and the failure shows up as a double charge or a repeated email long after the log line that caused it has scrolled away. The rule is short: any tool with a side effect gets a stable key derived from the business action, enforced by the remote where it can be and by your own dedup store where it cannot, with a reservation that survives concurrent attempts and a TTL that outlives your longest retry. Do that and your retry logic goes from a liability into exactly the safety net it was supposed to be.

If your agent has write tools and you have never traced what happens when one of them times out after the work committed, that is the incident waiting to happen. Book a consultation call and we can walk through which of your tools are safe to retry today and which ones are one dropped response away from a duplicate.