Your Agent Calls One Tool, Waits, Then Calls the Next

Here is the shape of almost every agent loop in production. The model returns a turn that asks for a tool. You run the tool, append the result, and send everything back. The model asks for another tool. You run that one, append, send back. Round and round until the model stops asking and writes its answer.

That loop is correct. It is also, for a whole class of tasks, needlessly slow, because it runs everything in single file even when the calls have nothing to do with each other.

Picture a travel agent answering "should I leave for the airport now." To answer, it needs the current traffic on the route, the flight's departure status, and the weather at the destination. Three lookups. None of them depends on the other two. In a naive loop the agent calls the traffic API, waits 400ms, calls the flight API, waits 500ms, calls the weather API, waits 300ms. The user waited 1.2 seconds, and for 1.2 seconds one machine sat idle waiting on another. Those three calls could have gone out at the same instant and come back in 500ms, the time of the slowest one alone. You left 700ms on the floor for no reason other than the shape of your loop.

The bottleneck moved, and most loops did not

For a couple of years the thing you waited on in an agent was the model. Inference was the slow part, and everyone optimized around token generation. That has quietly stopped being true for tool-heavy agents. The model turn is often the fast part now. What you actually wait on is tools: database queries, third-party APIs, retrieval calls, other services. And an agent that issues those tools one at a time pays the cumulative latency of every single one, in series, while each dependency sits idle waiting its turn.

The fix is not exotic, and it is not something you have to invent. The model already does the hard part for you.

The model tells you what is independent

Every major provider now supports what is usually called parallel function calling. When the model can see that several tool calls are independent, it does not dribble them out one per turn. It puts all of them into a single assistant turn. That travel agent's model, asked the airport question, returns one response containing three tool calls: traffic, flight, weather, together.

That grouping is a signal, and it is the whole game. The model is asserting that these calls do not depend on one another and can be issued at the same time. The only reason so many agents throw that signal away is that the standard execution loop iterates over the tool calls and awaits each one in turn, which serializes calls the model deliberately handed you as a batch.

Here is the loop that wastes it. This is the default in a lot of codebases because it is the obvious way to write it:

# The serial loop. It receives a batch of independent tool calls
# and then dutifully runs them one at a time, awaiting each before
# starting the next. The model grouped them; this ungroups them.
async def run_tools_serial(tool_calls, registry):
    results = []
    for call in tool_calls:
        fn = registry[call.name]
        output = await fn(**call.arguments)   # blocks here every time
        results.append({
            "tool_call_id": call.id,
            "content": output,
        })
    return results

If the model handed you three calls of 400, 500, and 300 milliseconds, this takes 1200ms. Every await is a full stop where nothing else happens.

Run the batch concurrently

The batch the model gave you is exactly the set of calls that are safe to run at once, so run them at once. In Python this is asyncio.gather; the equivalent exists in every runtime with async I/O.

import asyncio

# The concurrent loop. Every call in the model's batch is launched
# at the same time; we wait once, for all of them to finish. Total
# time is the slowest single call, not the sum of all of them.
async def run_tools_parallel(tool_calls, registry):
    async def invoke(call):
        fn = registry[call.name]
        output = await fn(**call.arguments)
        return {"tool_call_id": call.id, "content": output}

    # return_exceptions=True is not optional here. Without it, the
    # first tool to raise cancels the rest, and you have coupled
    # calls that were supposed to be independent.
    settled = await asyncio.gather(
        *(invoke(call) for call in tool_calls),
        return_exceptions=True,
    )
    return settled

Same three calls, 500ms total instead of 1200ms. Across a session with a lot of these turns, that is the difference between an agent that feels responsive and one that feels like it is thinking out loud. Published benchmarks put the wall-clock speedup somewhere between two and four times for tool-heavy workloads, and the exact number just depends on how many independent calls your tasks tend to produce and how uneven their latencies are.

The critical detail is return_exceptions=True. Leave it off and gather cancels every outstanding call the moment one of them raises. You would have taken three calls the model told you were independent and wired them together so that any one failing kills the other two. That is the opposite of what you want. With it on, you get back a list where each slot is either a result or the exception that call raised, and you deal with them one by one.

Feed partial failures back cleanly

Once the batch settles, the model is expecting a result for every tool_call_id it sent. It does not care whether a call succeeded; it cares that each one is accounted for. So translate a failed call into a normal-looking tool result whose content is a structured error, and hand the whole set back.

def to_tool_messages(tool_calls, settled):
    messages = []
    for call, outcome in zip(tool_calls, settled):
        if isinstance(outcome, Exception):
            # The failed slot still comes back as a tool result, so the
            # model can retry just this call or proceed with what it has.
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": f'{{"error": "{type(outcome).__name__}: {outcome}"}}',
            })
        else:
            messages.append({
                "role": "tool",
                "tool_call_id": outcome["tool_call_id"],
                "content": outcome["content"],
            })
    return messages

Now a batch where the flight API times out still returns useful traffic and weather, and the model can decide to retry the flight lookup alone or answer with a caveat. One flaky dependency degrades the answer instead of failing the turn. This is the same discipline as making tool calls recover instead of retrying forever, applied to a batch: each call owns its own fate.

Concurrency you do not bound will find your rate limits

There is a trap on the other side of this, and it is worth naming before you ship. The model does not know your infrastructure. If a task makes it emit forty independent calls, gather will happily launch all forty at once, and now you are hitting a third-party API forty connections deep, or opening forty database handles, or blowing straight through a rate limit that starts handing you 429s. You solved a latency problem by creating a load problem.

The fix is to cap how many calls run at once with a semaphore, so you get concurrency without an unbounded fan-out.

import asyncio

# Cap in-flight calls so a wide batch cannot stampede a dependency.
# Ten is a starting point; tune it to what the slowest downstream
# service can actually absorb.
async def run_tools_bounded(tool_calls, registry, limit=10):
    sem = asyncio.Semaphore(limit)

    async def invoke(call):
        async with sem:                       # waits if 'limit' are already running
            fn = registry[call.name]
            output = await fn(**call.arguments)
            return {"tool_call_id": call.id, "content": output}

    return await asyncio.gather(
        *(invoke(call) for call in tool_calls),
        return_exceptions=True,
    )

A batch of eight still runs eight-wide and finishes in the time of its slowest call. A batch of forty runs ten at a time, which is slower than a full stampede but far faster than pure serial, and it does not knock over the service you depend on. This is backpressure applied at the tool layer: the point is never maximum concurrency, it is the most concurrency your downstream can absorb without falling over.

The pitfalls that actually bite

Do not parallelize across turns on your own. The safety guarantee only holds inside a single model turn, because that batch is the model telling you these specific calls are independent. If you get clever and start firing off the next turn's likely calls before this turn resolves, you are now guessing at dependencies the model never promised, and a write that should have happened after a read happens before it. Concurrency within the batch the model gave you is free and safe. Concurrency you invent between turns is speculation, and it needs a completely different and more careful design.

Watch out for non-idempotent writes landing together. Independent reads are the easy, common case. If a batch ever contains two calls that mutate the same resource, running them concurrently makes their order nondeterministic, and last-write-wins races appear. Reads parallelize without a second thought. For writes, either confirm they touch different resources or keep the mutating ones serial.

Shared clients need to be concurrency-safe. The moment ten calls run at once, any HTTP client, database pool, or SDK object they share is being used from ten coroutines at the same time. Most modern async clients handle this; some older ones assume one caller at a time and corrupt state under concurrency. Check before you trust it, because this failure is intermittent and miserable to debug.

Streaming is the other half of the latency story. Parallel execution cuts the real wait. Streaming the model's final tokens as they generate cuts the perceived wait, because the user sees output start immediately instead of staring at a spinner. They are complementary. Do both: run the tools concurrently so the real latency is the slowest call, and stream the answer so the first words appear the instant the model has them.

The takeaway

An agent that runs tool calls one at a time is paying, on every multi-tool turn, for dead time that the model already told you how to eliminate. When a turn comes back with more than one tool call in it, that is not a list to loop over. It is a batch to launch. Run the batch concurrently, gather every result including the failures, cap the fan-out with a semaphore so a wide batch cannot flatten a dependency, and keep genuine writes serial. It is a change to one function in your agent loop, and on a tool-heavy agent it is often the single biggest latency win available, without touching the model, the prompt, or a line of your tools.

If your agents feel sluggish and you suspect the loop is spending its time waiting rather than thinking, this is exactly the kind of production latency work I help teams sort out. Book a consultation call and we can find where your agent is standing still.