Your Agent Answers the Same Question Fifty Times a Day

Pull the logs for any agent that has been in production for a month and group the incoming requests by meaning rather than by exact string. You will find the same handful of questions over and over. "What is your refund policy." "How do I reset my password." "Refund policy?" "how do i get a refund." Different words, one answer. Your agent embeds each one, reasons about it from scratch, and pays full price in tokens and latency to produce a response it has already produced dozens of times that day.

That is money on the floor, and it is slow. A live model call is seconds of latency and real tokens. The answer to "what is your refund policy" did not change between nine in the morning and nine oh two. You are paying to recompute a constant.

Semantic caching is the fix, and it is one of the highest-leverage things you can add to a mature agent. But it is also one of the easiest to get wrong in a way that quietly serves customers the wrong answer. I want to walk through how it works, how to build it so it actually helps, and the guardrails that keep a cache hit from becoming an incident.

Why a normal cache does not work here

Every engineer already knows caching. Key, value, time to live. The instinct is to hash the request and look it up. For LLM traffic that instinct fails on the first request, because natural language does not repeat exactly.

Here is the naive version, so we can see exactly where it breaks:

import hashlib

cache: dict[str, str] = {}

def cache_key(prompt: str) -> str:
    # Exact-match key. Fine for identical strings, useless for paraphrases.
    return hashlib.sha256(prompt.strip().lower().encode()).hexdigest()

def answer(prompt: str) -> str:
    key = cache_key(prompt)
    if key in cache:
        return cache[key]
    result = call_model(prompt)  # the expensive part
    cache[key] = result
    return result

This hits only when two users type the exact same characters. "What is your refund policy" and "what's your refund policy?" produce different hashes, so the cache misses and you call the model anyway. In real traffic the exact-match hit rate on free-text questions is close to nothing. The cache is technically working and practically doing almost no work.

The problem is that the thing you want to match on is meaning, not bytes. And meaning is exactly what embeddings encode.

The core idea: match on meaning, not on string

A semantic cache stores each answered question as a vector alongside its response. When a new question arrives, you embed it, search the cache for the nearest stored vector, and if that neighbor is close enough, you return its stored answer without ever calling the model.

This is the same retrieval machinery behind RAG, pointed at your own past answers instead of a document corpus. If you have already built retrieval, you have most of the parts. If you want the deeper version of the similarity-search mechanics, I went through them in fixing retrieval instead of blaming the model; here we only need the nearest-neighbor lookup and a threshold.

import numpy as np

class SemanticCache:
    def __init__(self, embed, threshold: float = 0.9):
        self.embed = embed          # text -> np.ndarray (normalized)
        self.threshold = threshold  # min cosine similarity to count as a hit
        self.vectors: list[np.ndarray] = []
        self.entries: list[dict] = []

    def get(self, prompt: str):
        if not self.vectors:
            return None
        q = self.embed(prompt)
        sims = np.array([float(q @ v) for v in self.vectors])  # cosine, vectors normalized
        best = int(sims.argmax())
        if sims[best] >= self.threshold:
            return self.entries[best]  # cache hit: known question, reuse the answer
        return None

    def put(self, prompt: str, response: str):
        self.vectors.append(self.embed(prompt))
        self.entries.append({"prompt": prompt, "response": response})

The lookup wrapper then looks almost like the naive one, except the hit condition is "close enough" instead of "identical":

def answer(prompt: str, cache: SemanticCache) -> str:
    hit = cache.get(prompt)
    if hit is not None:
        return hit["response"]        # skipped the model entirely
    result = call_model(prompt)
    cache.put(prompt, result)
    return result

In production you would not hold vectors in a Python list. You put them in whatever vector store you already run, or in Redis with its vector search, or Postgres with pgvector, so the lookup survives restarts and scales past memory. The logic is identical: embed, nearest neighbor, threshold check.

That is the whole happy path. The reason semantic caching has a reputation for being risky is everything the happy path leaves out.

The threshold is the entire game

Every decision in a semantic cache collapses into one number: how similar is similar enough. Set it too high and you get exact-match behavior with extra steps, almost no hits, no savings. Set it too low and you serve the answer to "how do I cancel my subscription" when the user asked "how do I pause my subscription," which are one word apart in embedding space and completely different in what the customer needs to hear.

That second failure is the one that hurts, because it is silent. There is no error. The user gets a fluent, confident, wrong answer, and your metrics show a fast cheap response. A cache that is too loose does not look broken. It looks great, right up until support tickets spike.

So do not pick the threshold by intuition. Instrument it. Log every lookup with its best-match score and whether you served the hit, then look at the distribution:

def get_with_logging(self, prompt: str, log):
    if not self.vectors:
        return None
    q = self.embed(prompt)
    sims = np.array([float(q @ v) for v in self.vectors])
    best = int(sims.argmax())
    score = float(sims[best])
    hit = score >= self.threshold
    log.record(
        prompt=prompt,
        matched=self.entries[best]["prompt"],
        score=round(score, 4),
        served_from_cache=hit,
    )
    return self.entries[best] if hit else None

Now you can pull the near-misses, the lookups that scored just under and just over your line, and read them with your own eyes. The pairs sitting at 0.88 tell you whether 0.90 is too strict. The pairs at 0.91 tell you whether it is too loose. This is the same discipline as putting LLM evals in CI: you are scoring a decision boundary against real cases instead of trusting a number you guessed. Build a small labeled set of "same question" and "different question" pairs from your logs, and the correct threshold stops being an argument and becomes a measurement.

Start at 0.9 on a strong embedding model and move only with that data in hand. Conservative and cheap beats loose and wrong.

Scope the key, or you will leak answers between users

The similarity match decides whether two questions mean the same thing. It says nothing about whether the same answer is correct for both askers. Those are different questions, and conflating them is how a cache serves user A's data to user B.

"What is my current plan" means the same thing no matter who types it. The answer does not. If your cache key is only the question text, two users asking that identical question hit the same cache entry, and the second one sees the first one's account details. The embedding was a perfect match. The result was a data leak.

The fix is that the cache key has to include every input that changes the correct answer, not just the words:

def scoped_key_fields(prompt: str, ctx: dict) -> dict:
    # Everything that changes the correct answer becomes part of the identity
    # of the cached entry. Same question + different tenant = different entry.
    return {
        "prompt": prompt,
        "tenant_id": ctx["tenant_id"],
        "locale": ctx["locale"],
        "tool_version": ctx["tool_version"],
    }

In practice you keep a separate cache namespace per tenant, or you filter the vector search by a metadata field so a lookup can only ever match entries from the same scope. General knowledge questions, the refund policy, how the product works, can live in a shared namespace and get the highest hit rates. Anything personalized gets scoped down or skipped. The rule is simple to state and easy to forget under deadline: if the answer depends on who is asking, the asker is part of the key.

This is the same instinct as the trust and isolation boundaries I wrote about in defending agents against prompt injection. Caching is another place where treating all requests as interchangeable is exactly the assumption that bites you.

Know what not to cache, and when to forget

Some answers should never enter the cache, and some should leave it on a schedule.

Do not cache anything time-sensitive or stateful. "How many orders shipped today" has a correct answer that changes by the hour. "What is the price of this item" changes when someone updates the catalog. If you cache those, you are not serving a fast answer, you are serving an old one. Route freshness-critical and side-effecting requests straight to the model, or better, straight to the actual source of truth.

For the things you do cache, staleness is a matter of when, not if. A refund policy is stable until the day legal rewrites it, and on that day every cached copy of the old policy is now wrong. So a time to live is not optional, and neither is targeted invalidation:

import time

class TTLSemanticCache(SemanticCache):
    def __init__(self, embed, threshold=0.9, ttl_seconds=3600):
        super().__init__(embed, threshold)
        self.ttl = ttl_seconds

    def get(self, prompt: str):
        hit = super().get(prompt)
        if hit is None:
            return None
        if time.monotonic() - hit["stored_at"] > self.ttl:
            return None  # expired: fall through to a fresh model call
        return hit

    def put(self, prompt: str, response: str):
        super().put(prompt, response)
        self.entries[-1]["stored_at"] = time.monotonic()

    def invalidate_topic(self, matcher):
        # Drop entries whose stored prompt matches, e.g. after a policy change.
        keep_v, keep_e = [], []
        for v, e in zip(self.vectors, self.entries):
            if not matcher(e["prompt"]):
                keep_v.append(v)
                keep_e.append(e)
        self.vectors, self.entries = keep_v, keep_e

Pick the TTL from how fast the underlying truth moves. Documentation answers can live for hours. Anything touching inventory or pricing lives for minutes if you cache it at all. And when a known event invalidates a class of answers, a policy update, a product rename, a config change, you want a way to drop exactly those entries without flushing the whole cache and losing every good hit you had warmed up.

The tradeoffs worth naming

A semantic cache is not free, and pretending otherwise is how it disappoints.

Every lookup costs one embedding call and one vector search before you even know if it is a hit. That is real work on the hot path. It is far cheaper than a full generation, so the math almost always favors caching once your hit rate clears a few percent, but on a low-repetition workload where users rarely ask the same thing, you can add latency and cost to chase savings that are not there. Measure your actual repetition rate before you build. If the same questions do not recur, this is not your optimization.

There is also an ongoing calibration cost. The threshold that was right at launch drifts as your traffic and your embedding model change. This is not set-and-forget infrastructure. It is a decision boundary that deserves the same monitoring as any other quality gate.

And caching is a cost lever, not a quality lever. It makes the answers you already give cheaper and faster. It does not make them better, and a loose one makes them worse. If your goal is spend and not correctness, it pairs naturally with model routing to cut AI costs: route the calls you cannot avoid to the cheapest capable model, and avoid the calls you can with a cache. Together they attack the bill from both sides.

The takeaway

Most agents answer the same small set of questions all day and pay full freight every single time. A semantic cache reclaims that, but the value is entirely in the guardrails, not the lookup. The lookup is twenty lines. The threshold you calibrate against real near-misses, the key you scope so answers cannot leak between users, and the invalidation you wire to the events that make answers stale, that is the part that decides whether the cache saves you money or quietly hands someone the wrong answer.

Build the boring version first. Conservative threshold, tightly scoped keys, aggressive expiry, full logging of every near-miss. Loosen it later with data in front of you, never on a hunch. A cache that is a little too strict costs you a few model calls. A cache that is a little too loose costs you a customer's trust.

If you are trying to get an agent's cost and latency under control without giving up on the quality of its answers, this is exactly the kind of tradeoff I help teams work through. Book a consultation call and we can look at where your token spend is actually going.