Your Background Agent Wakes Up More Than You Think

A team I was helping had a reporting agent that woke on a schedule, read a few dashboards, and posted a short summary to a channel. It ran quietly for weeks. Then someone added a webhook so the same agent would also run whenever a campaign config changed, and someone else wrapped the scheduler in a retry so a failed run would try again. Nothing inside the agent changed. One afternoon it posted the same summary four times in six minutes, each slightly different, and one of those runs kicked off a downstream export that was only ever supposed to happen once.

Nobody wrote a bug. Each piece did what it was told. The schedule fired, the webhook fired, the retry fired, and all of them pointed at an agent that had no idea it was being woken from three directions at once. This is the shape of the problem that arrives the moment an agent stops being something a user starts and becomes something that starts itself. The autonomous, always-on agents shipping this autumn, the ones meant to pursue a goal between conversations and act on their own schedule, make it the common case rather than the exception. The agent is fine. What is missing is the layer in front of it that decides whether a wake should happen at all.

A background agent has no turn boundary

An interactive agent has a natural frame around every run. A user sends a message, the agent works, the agent answers, the turn ends. That boundary does a lot of quiet work. It is where you start the clock, where you decide the request is still wanted, where you know the run is over. One request in, one answer out, and a human on the other end who notices when something is wrong.

A background agent has none of that. It is woken by a trigger, not a person, and the trigger carries no sense of whether now is a good time, whether a previous wake is still running, or whether this wake is a duplicate of one that fired a second ago. There is no user staring at a spinner who will give up and close the tab. A survey of always-on agents frames the hard part precisely: these systems carry durable state and trigger conditions across runs, and the literature has spent far more effort on accumulating that state than on governing when it should act and how to recover when a run goes wrong. The trigger condition is the part that decides when latent state becomes an action, and it is the part most systems leave implicit.

So the first move is to stop treating the trigger as a direct call into the agent. Put a layer between them. Every wake, from every source, becomes an event that the layer inspects before any model token is spent: is a run for this already in flight, have I seen this exact wake before, is there budget left to do it. The agent only runs when the layer says yes. Everything below is what that layer has to decide.

Coalesce the wakes before they fan out

The first decision is whether this wake should run at all given what is already running. Two patterns produce most of the duplicates. A scheduled agent overlaps itself: the interval is five minutes, a run takes seven, and now two runs of the same job are live at once, reading the same state and racing to write it. An event-driven agent bursts: an upstream system emits twenty events in a second because someone did a bulk edit, and twenty wakes stack up for work that only needed to happen once.

Workflow schedulers solved the overlap half of this years ago, and the vocabulary is worth borrowing. Temporal's scheduling overlap policies let a schedule skip a new run while one is still going, buffer it to run after, or cancel the old one in favor of the new. The point is that overlap is a decision you make deliberately, not an accident you discover in the logs. For a background agent, the same three choices apply, plus one more that events need: coalesce a burst of wakes that are all about the same thing into a single run.

import time

class Coalescer:
    def __init__(self, debounce_s=2.0):
        self.debounce_s = debounce_s
        self.running = set()      # keys with a run in flight
        self.pending = {}         # key -> earliest time it may run

    def admit(self, key: str) -> bool:
        now = time.monotonic()
        # Skip policy: a run for this key is already live, drop the wake.
        if key in self.running:
            return False
        # Debounce: hold a burst for a quiet window, so twenty events
        # about the same key collapse into one wake, not twenty.
        due = self.pending.get(key)
        if due is None:
            self.pending[key] = now + self.debounce_s
            return False
        if now < due:
            return False
        del self.pending[key]
        self.running.add(key)
        return True

    def finished(self, key: str):
        self.running.discard(key)

The key is doing the real work here, and choosing it well is the whole trick. It should name what the wake is about, not when it fired. For the reporting agent, the key is the campaign id, not the webhook delivery id, so ten edits to one campaign collapse into one run. Get the key too specific and nothing coalesces; get it too broad and unrelated work blocks each other. This is the same discipline as a guard that detects a stalled loop, applied one level up: instead of noticing a single run going nowhere, you notice the fleet of runs doing the same thing.

Make the wake idempotent

Coalescing handles wakes that arrive close together. It does nothing for the wake that arrives after a crash. A run claims a key, does half its work, takes an irreversible action, and then the process dies before it records that it finished. The coalescer forgets the key on restart. The next trigger fires, the key is free, and the agent does the whole thing again, including the action it already took. In-memory state does not survive the exact failure that causes duplicates.

The fix is the oldest trick in payments, and it is worth copying exactly. Stripe's idempotency keys record the result of an operation against a client-supplied key, and a retry with the same key returns the stored result instead of doing the work twice. They also compare the parameters of the retry against the original and reject a key reused with different arguments, which catches the subtle bug where a key is accidentally shared across unrelated requests. Apply the pattern to the wake itself, not just to the tool calls underneath it. This is the same idea as idempotency keys on side-effecting tools, lifted to the level of the whole run so a repeated trigger is caught before the agent even starts.

import json, hashlib

class WakeLedger:
    """Durable record of wakes, keyed by what the wake is about plus its inputs."""
    def __init__(self, store):
        self.store = store  # any durable kv: a table, redis with AOF, etc.

    def _id(self, key: str, inputs: dict) -> str:
        body = json.dumps(inputs, sort_keys=True)
        return key + ":" + hashlib.sha256(body.encode()).hexdigest()[:16]

    def claim(self, key: str, inputs: dict):
        wake_id = self._id(key, inputs)
        existing = self.store.get(wake_id)
        if existing and existing["state"] == "done":
            return None                      # already finished, do nothing
        if existing and existing["state"] == "running":
            return None                      # a live run owns it, do nothing
        self.store.put(wake_id, {"state": "running", "at": _now()})
        return wake_id                       # caller owns this wake

    def complete(self, wake_id: str, result: dict):
        self.store.put(wake_id, {"state": "done", "result": result, "at": _now()})

The ledger has to be durable, because its entire job is to survive the crash that the coalescer cannot. Write the claim before the work starts, write the completion after it ends, and a restart in the middle finds a running record it can either resume or reconcile rather than a clean slate it will happily repeat. For runs that take real steps and must pick up where they left off, this is the entry point to durable execution and checkpointing: the wake claim is the first checkpoint, and the agent's progress hangs off it.

Give every wake a budget and a deadline

An interactive run is bounded by a human's patience. A background run is bounded by nothing unless you say so, and that is how a quiet agent becomes a loud invoice. A wake that should read three dashboards instead gets stuck retrying a flaky tool, or talks itself into a twelve-step research detour, and because no one is watching, it runs until it finishes or falls over. Multiply by a trigger that fires more often than you planned and the bill is real.

Two bounds, at two levels. Each wake gets its own budget, measured in the things that actually cost money: tokens and tool calls, not just wall clock. And the fleet gets a rolling cap on how many wakes are allowed in a window, so a trigger storm cannot turn a modest per-run budget into a large total by firing a thousand times. The per-wake budget is the same discipline as an absolute deadline passed through the whole run; the rolling cap is where it meets per-tenant cost guardrails, because the tenant you are protecting the budget of is sometimes your own background fleet.

import time

class WakeBudget:
    def __init__(self, max_tokens, max_tool_calls, deadline_s):
        self.max_tokens, self.max_tool_calls = max_tokens, max_tool_calls
        self.at = time.monotonic() + deadline_s
        self.tokens = self.tool_calls = 0

    def charge_tokens(self, n):
        self.tokens += n
        self._check()

    def charge_tool_call(self):
        self.tool_calls += 1
        self._check()

    def _check(self):
        if self.tokens > self.max_tokens:
            raise BudgetExceeded(f"tokens {self.tokens} over {self.max_tokens}")
        if self.tool_calls > self.max_tool_calls:
            raise BudgetExceeded("tool-call budget spent")
        if time.monotonic() > self.at:
            raise BudgetExceeded("wake deadline passed")

The budget object travels with the run and every expensive step charges it before proceeding, so the run stops itself the moment it crosses a line rather than after. The rolling cap lives in the trigger layer, next to the coalescer, and it is the cheapest insurance you will buy: one counter that refuses the thousandth wake in an hour and pages you instead.

The supervisor: a kill switch for unattended runs

Everything so far bounds a run from the inside. The last piece bounds it from the outside, because the failure you cannot predict is the one the run does not notice. A run convinced it is making progress will spend its whole budget being wrong. A trigger whose upstream broke will fire in a tight loop. For those, you want a hand on a switch that is not held by the agent.

The supervisor is a small, dumb process that the agent does not control. It watches the wake ledger and the rolling counter, and it can do two things: refuse new wakes, and signal a live run to stop. Refusing new wakes is the circuit breaker: when wakes per minute cross a threshold, or the error rate on recent runs spikes, the layer stops admitting triggers and holds them, so the blast radius of a broken upstream is bounded to whatever was already in flight. Signaling a live run is the kill switch: a flag the budget object checks on every step, so an operator, or the supervisor itself, can tear a run down cleanly mid-flight.

class Supervisor:
    def __init__(self, ledger, max_per_min, max_error_rate):
        self.ledger, self.max_per_min, self.max_error_rate = ledger, max_per_min, max_error_rate
        self.halted = False

    def admit_wake(self) -> bool:
        if self.halted:
            return False                      # circuit open: hold all triggers
        if self.ledger.wakes_last_minute() > self.max_per_min:
            self.halted = True                # trip on a storm, page a human
            return False
        if self.ledger.recent_error_rate() > self.max_error_rate:
            self.halted = True                # trip on a run of failures
            return False
        return True

    def should_abort(self, wake_id) -> bool:
        return self.halted or self.ledger.is_killed(wake_id)

A halt is deliberately sticky. It does not clear itself when the numbers settle, because the thing that tripped it, a broken webhook, a model degradation, a bad deploy, is rarely fixed by waiting. A human resets it, which means a human has looked. For the genuinely consequential actions a background agent might take without anyone watching, the supervisor is not enough on its own, and the step belongs behind a human-in-the-loop approval gate so the irreversible thing never fires unattended in the first place.

Where this breaks

A few pitfalls, roughly in the order you will hit them.

The coalesce key is wrong. Key on the delivery id and nothing ever coalesces, because every webhook delivery is unique. Key on something too coarse, like a single global key, and unrelated work serializes behind one run. Spend the time to key on the thing the wake is actually about, and write that choice down, because the next person will add a trigger and reuse your key without knowing what it meant.

The ledger is not actually durable. An in-memory set or a cache with no persistence looks like it works in every test and fails on the one restart that matters, which is the restart in the middle of a run. If the ledger does not survive a crash, it is a coalescer, not an idempotency guard, and the duplicate you were trying to prevent is exactly the one that gets through.

Debounce adds latency you did not account for. Holding a burst for a quiet window means the agent reacts a few seconds late by design. That is fine for a reporting agent and wrong for anything that needs to act immediately. Set the debounce window per trigger, not globally, and keep it near zero for the triggers where a few seconds of delay is a real cost.

The budget counts tokens but not the subagents. A run that fans out into subagents or parallel tool calls has to charge all of them against the one wake budget, or each branch quietly gets the full budget and the total is a multiple of what you set. Thread the same budget object through the whole tree, the way you would a deadline, so the branches share one pool.

Nobody watches the supervisor. A circuit that trips silently is a background agent that has stopped doing its job, which is its own incident. The halt has to page someone, and the held triggers have to be visible, or you trade a loud failure for a quiet one, which is usually worse.

The takeaway

A background agent is not an interactive agent with the user removed. Taking the user out removes the turn boundary that was quietly doing the work of starting the clock, deciding the request was still wanted, and noticing when something went wrong. Put that boundary back as an explicit layer in front of the agent. Coalesce overlapping and bursty wakes so the same work does not run twice at once. Key dispatch durably by what the wake is about, so a trigger that fires again after a crash is caught before the agent starts. Budget every run in tokens, tool calls, and time, cap how often the fleet may wake, and give a supervisor a kill switch the agent cannot touch. None of it is much code. It is the difference between an agent that runs when it should and one that runs whenever anything twitches.

If you have an agent that now starts itself, on a schedule or a webhook, and you have never traced what happens when two triggers land at once or one fires after a crash, that is worth checking before it posts the same thing four times in front of a customer. Book a consultation call and we can map where your triggers fan out and where a run can act twice.