Your Agent Fetches the Same Row Ten Times a Session

Attach a counter to the tool layer of a long-running agent and watch what it actually calls. You will almost always find the same handful of reads repeated across a session: the same user record pulled on turn 2, turn 5, and turn 9, the same config fetched before every action, the same vector search reissued because the model circled back to a subtopic it already explored. Each of those is a full network round trip. You are paying the latency, the downstream quota, and the rate-limit budget every single time, for an answer that has not changed since the last time you asked.

This is dead cost, and it is the easiest kind to remove, because the fix is a pattern every backend engineer already knows: cache the read. The reason agents so often skip it is that a naive cache in front of tools is dangerous in a way a naive cache in front of a static endpoint is not. An agent that reads a stale value then acts on it does not just render an old number on a page, it takes a wrong action in the world. So the interesting engineering here is not the cache. It is the discipline around it: knowing which tools are safe to cache, bounding how stale an answer can get, and invalidating the instant something changes underneath you.

The business problem, in latency and dollars

Two costs stack up. The first is wall clock. A tool call that takes 300 milliseconds does not sound like much until an agent makes forty of them across a multi-step task and a third are exact repeats. That repeated third is pure dead time the user watches as a spinner, and on an interactive product every extra second measurably drops completion. The second is spend and load. Every duplicate call consumes a request against a metered API, adds load to a database, and eats into a rate limit you are sharing across a fleet. Run a hundred agents that all fetch the same popular record and you have turned one logical read into a hundred physical ones against a service that never asked for the traffic.

The frustrating part is that most of this is trivially avoidable. Tool results, unlike model outputs, are usually deterministic. Ask a database for row 42 twice in the same second and you expect the same bytes back. That determinism is exactly what a cache exploits, and it is why a tool result cache is a different and safer animal than a semantic cache. A semantic cache matches similar user questions to a stored answer and lives with the risk of a fuzzy match returning the wrong thing. A tool result cache keys on the tool name and the exact arguments, so a hit is an exact repeat, not a guess. And it is orthogonal to prefix caching, which cuts the model round trip rather than the tool round trip. The two compose: one removes token cost, the other removes I/O.

Step one: classify what is even cacheable

The first thing to build is not the cache. It is the policy that decides what may enter it. This has to be explicit, per tool, and default to off, because the failure mode of guessing wrong is serving a stale answer that changes an action. Treat cacheability as a property you declare when you register a tool, the same way you would declare its schema.

from dataclasses import dataclass
from typing import Callable, Awaitable, Any

@dataclass(frozen=True)
class CachePolicy:
    cacheable: bool            # off by default; must be opted in per tool
    ttl_seconds: int = 0       # upper bound on how stale a hit may be
    reads: tuple[str, ...] = ()   # entity families this read depends on
    writes: tuple[str, ...] = ()  # entity families this call mutates

@dataclass
class Tool:
    name: str
    run: Callable[[dict], Awaitable[Any]]
    policy: CachePolicy

# A pure read of one user's profile: safe to cache, bounded to 60s,
# and tagged so a write to the same user can find and clear it.
get_user = Tool(
    name="get_user",
    run=fetch_user_from_db,
    policy=CachePolicy(cacheable=True, ttl_seconds=60, reads=("user",)),
)

# A write: never cached, and it declares the family it invalidates.
update_user = Tool(
    name="update_user",
    run=write_user_to_db,
    policy=CachePolicy(cacheable=False, writes=("user",)),
)

The rule that keeps you out of trouble is that cacheable starts False and you turn it on only for tools that are genuinely read-only and idempotent. A search, a lookup, a GET. Anything that mutates state, sends a message, or spends money stays uncached forever, for the same reason you keep those calls deliberate rather than speculative. The ttl_seconds field is your honesty about staleness: it is the maximum age of any answer you are willing to act on. Sixty seconds for a user profile, five for a live inventory count, an hour for a rarely changing config. Pick it per tool from how fast the underlying data actually moves.

Step two: a cache key that means exactly one call

A cache is only correct if identical calls produce identical keys and different calls never collide. The trap is argument order and formatting: {"id": 42, "fields": ["name"]} and {"fields": ["name"], "id": 42} are the same call and must hash the same. Normalize before you key.

import hashlib, json

def cache_key(tool_name: str, args: dict) -> str:
    # sort_keys makes the encoding order-independent; separators strips
    # incidental whitespace. Same logical call, same bytes, same key.
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256(canonical.encode()).hexdigest()
    return f"tool:{tool_name}:{digest}"

Keep the key scoped to what actually determines the result. If a tool's output depends on the calling tenant, the tenant id belongs in the args you hash, or you will leak one tenant's cached read to another, which is a correctness and a privacy bug at once. When in doubt, put it in the key. An over-specific key costs you a few extra misses. An under-specific key serves the wrong data.

Step three: singleflight, so a fan-out is one request

Before caching a result you have to produce it, and this is where the second win hides. Agents fan out parallel tool calls, and fleets of agents hit the same popular lookups at the same instant. Without protection, ten concurrent requests for the same uncached key become ten real calls, and the cache does nothing because none of them has finished writing yet. Singleflight fixes this by letting the first caller run and making every other caller wait on that same in-flight result.

import asyncio

class Singleflight:
    def __init__(self) -> None:
        self._inflight: dict[str, asyncio.Future] = {}

    async def do(self, key: str, fn: Callable[[], Awaitable[Any]]) -> Any:
        existing = self._inflight.get(key)
        if existing is not None:
            # Someone is already computing this exact call. Wait on theirs.
            return await existing

        loop = asyncio.get_running_loop()
        fut: asyncio.Future = loop.create_future()
        self._inflight[key] = fut
        try:
            result = await fn()
            fut.set_result(result)
            return result
        except Exception as exc:            # losers see the same failure
            fut.set_exception(exc)
            raise
        finally:
            # Always clear, so the next call after this one runs fresh.
            self._inflight.pop(key, None)

Singleflight and caching are complementary, not the same thing. The cache removes repeats across time, the same call on turn 2 and turn 9. Singleflight removes repeats across concurrency, the same call issued by ten workers in the same millisecond. You want both, and singleflight matters most exactly when an agent runs tool calls in parallel and would otherwise multiply a shared read by the width of the fan-out.

Step four: the executor that ties it together

Now the pieces compose into one wrapper around tool execution. Read path: check the cache, honor the TTL, singleflight the miss, store the result. Write path: never cache, and on completion invalidate every cached read tagged with a family this write mutates.

import time

class CachingExecutor:
    def __init__(self, tools: dict[str, Tool]):
        self.tools = tools
        self.store: dict[str, tuple[float, Any]] = {}     # key -> (expiry, value)
        self.tags: dict[str, set[str]] = {}               # family -> keys
        self.sf = Singleflight()

    async def call(self, name: str, args: dict) -> Any:
        tool = self.tools[name]
        policy = tool.policy

        # Writes run immediately, then bust the reads they invalidate.
        if policy.writes:
            result = await tool.run(args)
            for family in policy.writes:
                for key in self.tags.pop(family, set()):
                    self.store.pop(key, None)
            return result

        if not policy.cacheable:
            return await tool.run(args)

        key = cache_key(name, args)
        hit = self.store.get(key)
        if hit is not None and hit[0] > time.monotonic():
            return hit[1]                                 # fresh cache hit

        # Miss (or expired): compute once, even under concurrent callers.
        async def produce() -> Any:
            value = await tool.run(args)
            self.store[key] = (time.monotonic() + policy.ttl_seconds, value)
            for family in policy.reads:
                self.tags.setdefault(family, set()).add(key)
            return value

        return await self.sf.do(key, produce)

The invalidation is the part that turns this from a liability into a safe optimization. Because every cached read is tagged with the entity families it depends on, a write can find and clear exactly the reads it affects. update_user runs, sees it writes the user family, and drops every cached get_user for that family in one pass. The next read misses and refetches live data. This is the same instinct behind idempotency keys for retry-safe side effects, pointed at reads instead of writes: make the cache correct by construction, so a stale value can never outlive the write that obsoleted it. If you want tighter invalidation, tag at the entity-id level rather than the family, so a write to user 42 clears only user 42's reads instead of every user read.

The tradeoffs, and where it bites

The obvious risk is staleness, and the TTL plus write-invalidation combination is what bounds it. But be honest that invalidation only covers writes that go through your agent. If some other system mutates the database out of band, your cache will happily serve the old value until the TTL expires, so keep TTLs short for anything with external writers and do not cache data that other services change unpredictably. The cache is a promise about freshness, and you can only keep the promise for changes you can see.

The second issue is scope. The in-memory version above is per-process, which is fine for a single agent worker but wrong the moment you scale horizontally, because each replica has its own cache and its own view of freshness. Move the store to a shared cache like Redis and two things change: hits are shared across the fleet, which is a bigger win, but invalidation now has to reach every replica, so the tag-to-key mapping lives in the shared store too and a write clears it for everyone at once. Singleflight also has to become distributed, or at least per-replica, to keep its guarantee under a real load.

The third is the subtle one: caching interacts with the model's sense of time. If an agent reads a value, acts, and reads again expecting to see its own effect, a cache that returns the pre-write value will confuse the loop badly. This is why write-invalidation has to be synchronous and has to run before the next read, and it is why you tag reads precisely. A cache that lags a write the agent just made produces exactly the kind of stale observation that sends an agent into a confused loop. Get the invalidation ordering right and the model never notices the cache exists, which is the whole point.

The takeaway

Agents repeat themselves, across turns and across a fleet, and most runtimes pay full price for every repeat. A tool result cache reclaims that spend, but only if you build it as a correctness feature rather than a speed hack. Classify tools so only genuine reads are cacheable and default everything else to off. Key on normalized arguments so a hit is an exact repeat, never a fuzzy guess. Add singleflight so a fan-out collapses to one request. And invalidate reads the instant a write touches the same data, so the cache can never outlive the truth. Do that and you cut latency and quota with no stale-data bugs, because the cache is bounded by a TTL you chose and cleared by every write that matters.

If your agents feel slow and expensive and you suspect a lot of that is the same reads over and over, it is usually measurable in an afternoon and fixable soon after. Book a consultation call and we can look at where your tool layer is repeating itself and whether a cache like this is the cleanest lever to pull.