Your Agent Is Idle Most of the Time It's Working

Watch a tool-using agent run with a stopwatch and profiler attached, and the first surprising thing is how little of the time the model is actually thinking. A retrieval-heavy agent might spend two seconds generating tokens and five seconds sitting on its hands, blocked on a vector search, a database query, an API call to some service three hops away. The model finishes its reasoning, emits a tool call, and then everything stops while a network round trip happens. Nothing is computing. The GPU is idle, the user is watching a spinner, and the wall clock keeps running.

This is the same problem a CPU faces every time it hits a branch. It does not know which way the code will go, so it could stall the whole pipeline waiting to find out. Instead it guesses, runs ahead on the predicted path, and throws away the work if the guess was wrong. That gamble is why your processor is fast. Agents can play exactly the same game, and most of them are leaving the speedup on the floor.

Where the time actually goes

Start by being honest about the shape of your latency, because speculative execution only helps if a real fraction of your wall clock is spent waiting on tools rather than generating tokens. For a chatbot that answers from the model's own weights, there is nothing to speculate on and you should stop reading. For an agent that plans a step, calls a slow tool, reads the result, plans the next step, and calls another slow tool, the waiting dominates, and that waiting is pure dead time you are already paying for.

The business cost is not abstract. Every extra second of latency on an interactive agent measurably drops completion rates, and for a voice agent it is worse, because silence past about a second reads as the system being broken. On the batch side, if you are running agents at scale, wall-clock time is throughput, and throughput is the size of the fleet you have to pay for. Cutting the idle time is not a nicety. It changes your unit economics.

The reason the time is recoverable is that the model's next move is often not a surprise. An agent that just searched a knowledge base is very likely to fetch the top document next. An agent that looked up a user is very likely to pull that user's recent orders. The trajectory so far is a strong predictor of the next tool call, and if you can predict it, you can start running it before the model asks.

The core idea: predict, prefetch, commit or discard

The mechanism has three parts. A speculator watches the trajectory and predicts the next tool call. An executor runs that predicted call in the background, concurrently with the model's next round of token generation. And a commit step, when the model finally emits its real call, checks whether the real call matches the speculation. On a match you return the cached result instantly and skip the round trip entirely. On a miss you throw the speculative result away and fall back to running the real call normally, exactly as you would have without any of this.

The critical property is that a wrong guess costs you nothing in correctness. You never show the user a speculative result and you never let it change the conversation. It is a cache that either has the answer or does not. The only cost of a miss is the wasted background work, which is why the tools you speculate on have to be cheap enough and, above all, safe enough to run for nothing.

Here is the safety gate, which is the first thing to build, not the last. Speculation runs calls that may be unnecessary, so it must only ever touch tools that are free of side effects or safely idempotent.

from dataclasses import dataclass
from typing import Callable, Awaitable

@dataclass
class Tool:
    name: str
    run: Callable[[dict], Awaitable[dict]]
    # Only tools flagged here are eligible for speculation. A read, a
    # search, a lookup: safe to run even if the model never asks for it.
    # Anything that writes, charges, sends, or books stays False. This
    # defaults closed on purpose, so a new tool is never speculated by
    # accident.
    speculatable: bool = False

def is_speculatable(tool: Tool) -> bool:
    return tool.speculatable

If you take one thing from this post, take that default. A speculative search that runs for nothing wastes a few milliseconds and some quota. A speculative refund that runs for nothing is an incident. The allowlist is not optional and it does not belong in a config file someone can flip without thinking. It belongs next to the tool definition, where the person adding a write tool has to make a conscious decision.

Building the speculator

The predictor can be as simple or as clever as your traffic justifies. The cheapest useful version is a trajectory cache: remember, across past runs, what tool call tended to follow a given state, and replay that guess. No extra model, no extra tokens, and it captures the boring-but-common patterns that make up most of a workload.

from collections import defaultdict

class TrajectoryCache:
    """Learns 'after state X, the next call was usually Y' from history."""

    def __init__(self) -> None:
        # key: a hash of the recent trajectory. value: counts per next call.
        self._counts: dict[str, dict[tuple, int]] = defaultdict(
            lambda: defaultdict(int)
        )

    def key(self, trajectory: list[dict]) -> str:
        # Use only the last couple of steps as the prediction context.
        # Too much history and every state looks unique, so nothing ever
        # matches and the cache never fires.
        recent = trajectory[-2:]
        return "|".join(f"{s['tool']}:{s.get('status','')}" for s in recent)

    def record(self, trajectory: list[dict], next_call: dict) -> None:
        sig = (next_call["tool"], _freeze(next_call["args"]))
        self._counts[self.key(trajectory)][sig] += 1

    def predict(self, trajectory: list[dict]) -> dict | None:
        options = self._counts.get(self.key(trajectory))
        if not options:
            return None
        (tool, frozen_args), hits = max(options.items(), key=lambda kv: kv[1])
        # Only bother if this path is actually common. A single past
        # occurrence is noise, not a prediction.
        if hits < 3:
            return None
        return {"tool": tool, "args": dict(frozen_args)}

def _freeze(args: dict) -> tuple:
    return tuple(sorted(args.items()))

For higher hit rates you can swap the cache for a small, fast model that reads the trajectory and drafts the next call, the same way speculative decoding uses a small draft model to propose tokens for a big one to verify. That buys accuracy at the cost of a little latency and spend on the draft, and whether it pays off depends on how much your tools cost you in wait time. Start with the cache. Reach for the draft model only when you have measured that the cache's hit rate is leaving real time on the table.

Wiring it into the loop

Now the executor. While the main model generates its next step, kick off the predicted call in the background. When the real call arrives, reconcile.

import asyncio

async def agent_step(model, tools, trajectory, cache):
    # 1. Predict and launch speculation BEFORE we ask the model anything.
    #    This background task runs during the model's generation time,
    #    which is the whole point: the wait is now overlapped, not serial.
    speculation = None
    guess = cache.predict(trajectory)
    if guess and is_speculatable(tools[guess["tool"]]):
        speculation = (guess, asyncio.create_task(
            tools[guess["tool"]].run(guess["args"])
        ))

    # 2. Meanwhile, the model decides its actual next call.
    real_call = await model.next_tool_call(trajectory)

    # 3. Commit or discard.
    if speculation:
        guess, task = speculation
        hit = (guess["tool"] == real_call["tool"]
               and guess["args"] == real_call["args"])
        if hit:
            result = await task          # already running or done: near-free
            cache.record(trajectory, real_call)
            return real_call, result, "hit"
        task.cancel()                    # wrong guess: drop the wasted work
        await asyncio.gather(task, return_exceptions=True)

    # 4. Miss (or nothing to speculate): run the real call the normal way.
    result = await tools[real_call["tool"]].run(real_call["args"])
    cache.record(trajectory, real_call)
    return real_call, result, "miss"

The whole trick lives in the ordering. The speculative task is created before the await on the model, so the tool round trip and the token generation happen at the same time instead of one after the other. On a hit, await task returns almost immediately because the work has been running the entire time the model was thinking. That overlap is the latency you get back, and it composes cleanly with parallel tool execution: parallelize the batch the model already gave you, and speculate on the batch you think is coming next.

The tradeoffs, and the ways it bites

The obvious cost is wasted work on a miss. Every wrong guess is a tool call you paid for and threw away, in quota, in load on the downstream service, and in your own rate-limit budget. This is why hit rate is the number to watch. Below roughly a coin flip, the wasted calls start to outweigh the saved time and you are just adding load for nothing. Instrument the hit rate per trajectory pattern and only speculate on the patterns where you are actually right most of the time. A predictor that knows when to stay quiet beats one that always guesses.

The subtle cost is load amplification on the tools you speculate against. If a hundred agents all speculate the same popular search, that downstream service now sees extra traffic it never used to, and speculative reads look identical to real ones from its side. Make sure the tools on your allowlist can absorb it, and consider a short-lived result cache in front of them so identical speculative and real calls collapse into one actual request.

Then there is the safety boundary, which is worth repeating because it is the one that turns a performance win into an outage. Never speculate a tool with side effects. If you feel tempted to speculate a write because it is on the hot path, the answer is not to relax the allowlist, it is to split the tool: a safe read that you speculate freely, and a separate committed write that only ever runs on the model's real, deliberate call. This is the same discipline behind idempotency keys for retry-safe side effects, applied one layer earlier. Keep reads speculative and writes deliberate and the whole scheme stays boring, which is exactly what you want from something running ahead of your agent's decisions.

One more, easy to miss: a speculative call can fail, and its failure must not leak. If the prefetch throws, swallow it, log it, and fall through to running the real call normally when the model asks. A speculation that turns its own error into the agent's error has broken the one promise the design makes, that a wrong or failed guess never changes the outcome.

The takeaway

Most agents are slow not because the model is slow but because the runtime is naive, running plan and fetch strictly one after another when the fetch was predictable all along. Speculative tool execution treats the next tool call the way a CPU treats a branch: guess it, run ahead, commit on a match, discard on a miss, and never let a wrong guess touch correctness. The engineering that matters is not the predictor, which can start as a three-line trajectory cache. It is the safety gate that keeps speculation to side-effect-free tools and the instrumentation that keeps you honest about your hit rate. Get those two right and you claw back the dead time your agent is already paying for.

If your agents feel sluggish and you suspect most of that time is spent waiting rather than thinking, that is usually measurable in an afternoon and fixable soon after. Book a consultation call and we can profile where your loop is idling and whether speculation is the lever worth pulling.