Your Agent Fans Out Faster Than Anything Downstream Can Take

Give an agent a goal broad enough and it will happily plan its way into a denial-of-service attack on your own infrastructure. Ask it to "enrich these 400 accounts," and a reasonable plan is to spawn a sub-task per account, and each sub-task fires three or four tool calls, and suddenly the run wants to make fifteen hundred outbound calls right now, all at once, against an API that tops out at ten requests a second and a model provider that starts returning 429 the moment you cross your quota.

Nothing in the agent stops this. The planner produced work, the executor tried to run all of it, and the only thing standing between you and a runaway fan-out was a Promise.all that scheduled everything the instant it could. The queue of pending work is unbounded, so it grows until it eats memory. The token budget is unbounded, so it spends until the run finishes or the invoice does. And the downstream tool you depend on gets hit so hard it starts timing out, which your retry logic reads as "try again," which adds more load. This is the failure mode nobody designs for and everybody hits the first time an agent gets ambitious in production.

The fix is not a smarter prompt. It is the same set of controls that keep any high-throughput system from destroying itself: bound the work, limit the concurrency, rate-limit the dependencies, and break the circuit when something downstream falls over. Agents just make the need urgent, because unlike a normal service they generate their own load dynamically and have no built-in sense of how much is too much.

The four things that are actually unbounded

When an agent fan-out melts down, it is usually because four separate quantities were left with no ceiling, and each one needs its own control.

The work queue is unbounded. A plan that spawns sub-tasks, some of which spawn more sub-tasks, produces a tree of pending work with no cap on its size. You need admission control: a bounded queue that refuses or defers work once it is full.

The concurrency is unbounded. Even with a fixed amount of work, firing all of it at once is what overwhelms a downstream tool. You need a limit on how many calls are in flight at the same time.

The request rate is unbounded. Concurrency limits how many run together, but a tool or model provider cares about requests per second over time, not just simultaneity. You need a rate limiter, and a token bucket is the standard shape.

The token budget is unbounded. Every LLM call and every sub-agent spends tokens, and a fan-out multiplies that spend fast. You need a budget that is charged as work runs and that a sub-agent cannot blow through on its own.

Handle all four and a runaway plan degrades into a slow plan instead of an incident. Handle none and you are one ambitious goal away from a bad afternoon.

Bound the concurrency and the queue first

The single highest-leverage change is to stop firing everything at once. A semaphore caps how many calls run concurrently, and a bounded queue caps how much work is allowed to wait. Together they turn an infinite fan-out into a steady stream.

import asyncio

# At most 8 tool calls run at once; at most 100 wait their turn.
sem = asyncio.Semaphore(8)
queue: asyncio.Queue = asyncio.Queue(maxsize=100)

async def submit(task) -> bool:
    # Admission control: if the queue is full, we shed load instead of
    # letting it grow without bound. Returning False lets the caller
    # defer, retry later, or tell the user we are at capacity.
    try:
        queue.put_nowait(task)
        return True
    except asyncio.QueueFull:
        return False

async def worker(run_tool):
    while True:
        task = await queue.get()
        try:
            async with sem:            # never more than 8 in flight
                await run_tool(task)
        finally:
            queue.task_done()

The important line is the except asyncio.QueueFull. That is backpressure. When the agent generates work faster than the workers can drain it, submit starts returning False, and the planner has to react to that signal instead of pretending the work vanished into an infinite buffer. A queue with no maxsize is not a queue, it is a memory leak with good intentions.

Pick the semaphore size from the slowest thing you call, not the fastest. If the tightest downstream tool handles ten concurrent requests before it degrades, eight is a sane cap even if the model provider could take far more.

Rate-limit with a token bucket

Concurrency caps how many calls overlap, but many providers enforce a rate: requests or tokens per second, measured over a window. Two calls that never overlap can still trip a rate limit if they happen close enough together. A token bucket is the clean way to enforce a sustained rate while still allowing short bursts.

import asyncio
import time

class TokenBucket:
    def __init__(self, rate: float, capacity: float):
        self.rate = rate          # tokens refilled per second
        self.capacity = capacity  # max burst you allow
        self.tokens = capacity
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self, cost: float = 1.0) -> None:
        async with self.lock:
            while True:
                now = time.monotonic()
                # Refill based on elapsed time, capped at capacity.
                self.tokens = min(
                    self.capacity,
                    self.tokens + (now - self.updated) * self.rate,
                )
                self.updated = now
                if self.tokens >= cost:
                    self.tokens -= cost
                    return
                # Not enough tokens: sleep exactly long enough to earn them.
                await asyncio.sleep((cost - self.tokens) / self.rate)

Call await bucket.acquire() right before each request and the bucket paces you to the configured rate no matter how much work the agent throws at it. The cost argument matters more than it looks: for a model provider that limits on tokens per minute rather than requests, pass the estimated token count as the cost, so one big completion draws down the bucket the way it draws down your real quota. A per-request limiter that ignores payload size will happily wave through one enormous call that blows the token limit on its own.

Break the circuit when a dependency is failing

Rate limiting protects a healthy dependency from too much load. A circuit breaker protects you from a dependency that is already broken. When a tool starts timing out or erroring, the worst thing you can do is keep calling it, because every doomed call ties up a concurrency slot and a chunk of your timeout budget while returning nothing. A breaker notices the failures, opens, and fails fast for a cooldown period instead of retrying into a wall.

import time

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    def __init__(self, fail_max: int = 5, reset_after: float = 30.0):
        self.fail_max = fail_max        # consecutive failures before opening
        self.reset_after = reset_after  # seconds to wait before a trial call
        self.failures = 0
        self.opened_at: float | None = None

    def _state(self) -> str:
        if self.opened_at is None:
            return "closed"
        if time.monotonic() - self.opened_at >= self.reset_after:
            return "half_open"   # allow one trial call through
        return "open"

    async def call(self, make_call):
        if self._state() == "open":
            # Fail fast, cheaply, without touching the sick dependency.
            raise CircuitOpen("downstream is failing; skipping the call")
        try:
            result = await make_call()
        except Exception:
            self.failures += 1
            if self.failures >= self.fail_max:
                self.opened_at = time.monotonic()  # trip the breaker
            raise
        # Success resets everything, including a half-open trial that passed.
        self.failures = 0
        self.opened_at = None
        return result

The half-open state is what makes this self-healing. After the cooldown, one call is allowed through as a probe. If it succeeds, the breaker closes and normal traffic resumes. If it fails, the breaker trips again for another cooldown. This is also where retries and rate limits meet: a plain retry loop with no breaker is exactly what turns a provider's brief 429 into a sustained storm, because every failure immediately generates more load. If you are building the retry side of this, I went through classifying and recovering from tool failures in when your agent keeps retrying a call that will never work. Back off with jitter, honor Retry-After, and let the breaker cut off the dependency once it is clearly down.

Give the fan-out a budget it cannot exceed

The last unbounded quantity is money. Concurrency and rate limits control how fast you spend, but not how much. A hierarchical token budget puts a hard ceiling on the whole run and hands each sub-agent a slice it cannot overspend, so one greedy branch cannot drain the pool the others need.

class BudgetExceeded(Exception):
    pass

class Budget:
    def __init__(self, total_tokens: int):
        self.remaining = total_tokens

    def charge(self, tokens: int) -> None:
        if tokens > self.remaining:
            raise BudgetExceeded(
                f"needs {tokens}, only {self.remaining} left"
            )
        self.remaining -= tokens

    def child(self, grant: int) -> "Budget":
        # Carve a sub-budget for a spawned sub-agent. The parent debits it
        # up front, so the child works within a ceiling and cannot reach
        # back into the shared pool no matter how it fans out.
        self.charge(grant)
        return Budget(grant)

Charge the budget after each call using the real token counts from the provider response, and check it before spawning a sub-agent. When a sub-agent hits BudgetExceeded, that is not a crash, it is the system correctly refusing to spend money it does not have. This is the same instinct as the orchestrator handing bounded work to workers, which I covered in moving from a monolithic agent to orchestrator and workers; the budget is just the resource version of that boundary. It turns "the agent spent $340 before someone noticed" into "the agent stopped at its cap and reported what it could not finish."

The tradeoffs worth naming

None of this is free, and each control has a failure mode of its own.

Bounds add latency and rejection. A bounded queue means that sometimes work is refused, and you have to decide what that means for the user: defer it, degrade gracefully, or surface a clear "at capacity" instead of hanging forever. Silently dropping shed work is its own bug, and it is worse than the meltdown because it looks like success.

Limits set too tight strangle throughput. A semaphore of two and a token bucket dripping one request a second will keep you safe and make the agent unusably slow. These numbers are not guesses to set once. They come from the real limits of the systems you call, and the right move is often adaptive concurrency, where you raise the limit while calls succeed and cut it hard the moment you see timeouts or 429s, the same additive-increase, multiplicative-decrease shape that TCP uses.

Circuit breakers can mask a real outage. A breaker that trips and stays quiet keeps your agent alive while a dependency is down, which is good, but if nothing watches the breaker state, you have hidden an outage instead of fixing it. Breaker open, queue depth, tokens spent, and shed count are the metrics that tell you the controls are working, and they belong on a dashboard, not buried in logs.

And these controls guard against runaway fan-out, not against a run that dies partway through. If your concern is a long run losing all its progress on a crash, that is a different problem, and I wrote about it separately in durable execution and checkpointing for agents. Backpressure keeps the run from overwhelming its dependencies. Durable execution keeps it from starting over.

The takeaway

An agent is a load generator that decides for itself how much load to create, which is exactly why it needs the controls a normal service takes for granted. The failure is never one dramatic bug. It is four quantities left unbounded at the same time: the work queue, the concurrency, the request rate, and the token budget. Cap each one and an over-ambitious plan slows down instead of falling over.

Start with the two that stop the bleeding: a bounded queue so work cannot pile up without limit, and a concurrency cap so you never fire everything at once. Add a token bucket to pace the dependencies that measure you over time, a circuit breaker so a sick tool fails fast instead of dragging every call down with it, and a hierarchical budget so no branch can spend the whole run. That is a few hundred lines, and it is the difference between an agent you can point at four hundred accounts and one you have to babysit.

If your agents are starting to fan out and you have already felt the first rate-limit storm or the first surprise invoice, this is exactly the kind of production reliability work I help teams get right. Book a consultation call and we can look at where your fan-out is spending money and load you did not intend.