You Pay Full Price for the Same 8,000-Token Prompt on Every Agent Step

You give your agent an 8,000-token system prompt. It explains the tools, the house style, the safety rules, the output format. Then you attach a dozen tool schemas, another few thousand tokens. The agent starts working: it calls a tool, reads the result, calls another, reads that, and so on for thirty steps.

On every one of those steps you send the whole thing again. The 8,000-token system prompt. The tool schemas. The entire conversation so far. That is how the API works, it is stateless, so each call carries the full context. And unless you have done something about it, you are paying full input price for those same tokens thirty times over, and the model is re-reading them from scratch every time before it produces a single new token.

That re-reading is not free, and it is not fast. The prefill phase, where the model ingests your prompt, is often the slowest part of a request. For an agent whose prompt barely changes between steps, you are burning money and latency on work the provider already did a second ago.

Prompt caching fixes exactly this, and most teams either do not use it or use it in a way that quietly does nothing. This is the difference between an agent that costs a few cents a run and one that costs a few dollars.

The business problem: the prefix is huge and the diff is tiny

Look at what actually changes between two consecutive agent steps. The system prompt is byte-for-byte identical. The tool schemas are identical. The first twenty-nine messages of the conversation are identical. The only new content is the latest tool result and the model's next move. You are resending fifteen thousand stable tokens to append two hundred new ones.

Providers noticed this shape and built for it. When you mark part of your prompt as cacheable, the provider saves the internal state from processing those tokens, the key/value tensors from the attention layers, and keys them by the exact bytes of the prefix. The next request that starts with the same bytes reuses that saved state instead of recomputing it. You get billed for the cached span at roughly a tenth of the normal input rate, and the prefill for that span mostly disappears.

The numbers are not subtle. A cache read costs about 0.1x the base input price. On an agent that resends a large prefix across dozens of steps, that is the bulk of your input tokens dropping to a tenth of their cost. Latency on a cache hit typically falls by a large margin too, because prefill was the expensive part and you just skipped most of it. One well-known case study moved dynamic content out of the wrong place in the prompt and took their hit rate from 7 percent to 84 percent, cutting overall LLM spend by well over half.

This is distinct from reusing whole answers with semantic caching. That technique looks at an incoming question, finds a past question that is close enough, and returns the old answer without calling the model at all. It changes what the user gets back, and a loose threshold serves the wrong answer. Prompt caching never touches the output. It reuses the model's own scratch work for tokens it has already seen, and the response is identical to the uncached one. It is a pure cost and latency lever with no quality risk, which is why it should be the first thing you reach for.

The one rule everything follows from

Prompt caching is a prefix match. The cache key is derived from the exact bytes of your prompt up to each cache breakpoint. Any change anywhere in the prefix, a single character, invalidates the cache for everything after it.

That one sentence explains every caching bug you will ever hit. Render order matters: the provider assembles your request as tools, then system, then messages. So the stable, never-changing content has to physically come first, and the volatile per-request content has to come last, after your cache breakpoint. Get the ordering right and caching mostly works for free. Get it wrong and no amount of cache markers will save you.

Here is the mistake almost everyone makes first. They build a helpful system prompt that includes the current time.

# BROKEN: the timestamp changes every request, so the prefix never matches.
from datetime import datetime, timezone

def system_prompt():
    now = datetime.now(timezone.utc).isoformat()
    return f"You are a support agent. The current time is {now}.\n\n{RULES}"

That timestamp sits at the very front of the prefix. Every request produces different leading bytes, so the cache is invalidated before it ever helps. You added a cache breakpoint downstream, you see cache_read_input_tokens sitting at zero, and you cannot work out why. The fix is to freeze the system prompt and inject anything volatile later, in the message stream, where it invalidates nothing ahead of it.

# WORKS: stable system prompt, marked cacheable. Volatile data moves into messages.
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": RULES,  # frozen: no timestamps, no user id, no request id
            "cache_control": {"type": "ephemeral"},  # breakpoint at end of system
        }
    ],
    messages=[
        # The current time is per-request context, so it belongs here, after the
        # cached prefix, not baked into the system prompt.
        {"role": "user", "content": f"[time: {now}] Where is my order?"},
    ],
)

The breakpoint on the last system block caches the tools and the system prompt together, because tools render before system. Everything up to that marker is now a reusable prefix. The per-request question, which genuinely differs every time, sits after it and costs full price, which is fine, because it is small.

Structuring the whole prompt: stable first, volatile last

Classify every input by how often it changes and lay the prompt out in that order:

• Never changes (the frozen system prompt, the tool schemas): put it first, before any breakpoint.
• Changes per session (a user profile, a retrieved document set for this conversation): put it after the global prefix, cache it per session.
• Changes per request (the current question, a timestamp, a fresh ID): put it dead last, after the final breakpoint, and do not cache it.

The classic layout for a retrieval-augmented call is: system instructions, then static few-shot examples, then the document context, then, at the very end, the user's actual question. If you accidentally put the varying question in the middle, every request writes a brand new cache entry and reads nothing, and you pay the cache-write premium for zero benefit.

A few silent invalidators to grep for in any code that builds the prefix. Each one changes the leading bytes and quietly kills your hit rate:

• datetime.now() or time.time() in the system prompt.
• uuid4() or a request ID placed early in the content.
• json.dumps(obj) without sort_keys=True, or iterating a set, which serializes in a nondeterministic order.
• A user or session ID interpolated into the system prompt, which fragments the cache per user and kills all cross-request sharing.
• A tool list built per user, so the tools block at position zero differs every time.

The agent-specific trap: do not mutate the history you already sent

For a single call, structuring the prompt is the whole job. For an agent, there is a second discipline, and it is where multi-step runs fall apart.

An agent grows its message list turn by turn. The right move is to append the new turn and put a cache breakpoint on the last block of it. Each subsequent request then reuses the entire prior conversation as a cached prefix, and the hits accumulate as the run gets longer.

# Roll the breakpoint forward onto the newest turn each step.
messages.append({"role": "user", "content": tool_results})
messages[-1]["content"][-1]["cache_control"] = {"type": "ephemeral"}

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    system=[{"type": "text", "text": RULES, "cache_control": {"type": "ephemeral"}}],
    tools=TOOLS,          # deterministic order, same every step
    messages=messages,    # everything before the newest turn is unchanged
)

The trap is any code that reaches back and edits earlier messages. Re-summarizing an old turn in place, renumbering things, injecting a fresh timestamp into message five on step twenty, all of these change bytes inside the already-cached prefix and invalidate everything after the edit. The moment you rewrite history, you pay full price to reprocess the whole conversation from the edit point forward. Treat the sent history as append-only. If a run gets long enough that you need to shrink it, do that through deliberate context compaction that produces a new stable prefix, not through in-place edits that thrash the cache every step.

There is one specific version of this worth calling out: injecting a mid-run instruction. You learn something partway through the run, a mode toggled, the user sent async context, and you want to tell the model. The wrong fix is to edit the top-level system prompt, which sits at the front of the prefix and invalidates the entire conversation. On recent models you can instead append a system-role message to the end of the message list. It carries operator authority and sits after the cached history, so nothing ahead of it is invalidated.

# Add an instruction mid-run without touching the cached system prompt.
messages.append({
    "role": "system",
    "content": "Auto-approve mode is on. Do not ask for confirmation on reads.",
})

One more agent-specific detail that bites people: each breakpoint only looks back a limited number of content blocks to find a matching cache entry. If a single agent turn appends more than that, common in a loop that fires many tool calls at once, the next request's breakpoint cannot see the previous cache and silently misses. The fix is to drop an intermediate breakpoint inside long turns so each one stays within the lookback window of the last cached block. You get four breakpoints per request; spend them on the boundaries that matter.

Measure it, because "I added cache_control" is not the same as "it is caching"

The cache is invisible until you read the usage numbers. Every response tells you exactly what happened:

u = response.usage
print("written to cache:", u.cache_creation_input_tokens)  # paid ~1.25x, once
print("served from cache:", u.cache_read_input_tokens)     # paid ~0.1x
print("uncached:", u.input_tokens)                          # paid full price

# The single most useful number for an agent run.
total_prefix = u.cache_read_input_tokens + u.cache_creation_input_tokens + u.input_tokens
hit_rate = u.cache_read_input_tokens / total_prefix if total_prefix else 0.0
print(f"cache hit rate: {hit_rate:.0%}")

Watch this across a run. On the first step you expect a big cache_creation_input_tokens number, that is the write, and it costs a bit more than a normal read, around 1.25x for the default short-lived cache. On every step after that you want to see cache_read_input_tokens carrying most of your prefix and input_tokens down to just the new turn. If cache_read_input_tokens stays at zero across steps that share a prefix, you have a silent invalidator. Diff the exact rendered prompt bytes between two consecutive requests and it will jump out at you.

Note also that input_tokens is only the uncached remainder. If your agent ran for ten minutes and that number reads four thousand, do not panic that you barely used any context. Add all three fields together to see the real prompt size; the rest was served from cache.

The tradeoffs worth naming

Caching is close to free money, but there are real edges.

Writes cost more than reads. Priming the cache costs about 1.25x a normal request for the default cache, and about 2x if you opt into the longer-lived one. That means caching only pays off when you actually get reads. For the default short-lived cache you break even after roughly the second request; for the longer-lived one you need a few. If a prompt genuinely changes from the first token every time, do not cache it, you would only ever pay the write premium and never collect a read.

The cache expires. The default entry lives for about five minutes after its last use. For a tight agent loop that is plenty, the next step lands well inside the window. For bursty or slow traffic with long gaps between requests, the entry evaporates between calls and every request pays a cold write. Either switch to the longer time-to-live, at the higher write cost, or pre-warm the cache at the start of a session so the first real request is already a hit.

The prefix has a minimum size. Below a model-dependent threshold, usually on the order of a thousand tokens or a couple thousand for some models, a prefix will not cache at all, silently, with no error. You just see a write count of zero. For agents this is rarely a problem because the prefixes are large, but it is why caching a tiny system prompt does nothing.

Caches are scoped to the model. Switch models mid-conversation and you lose the cache entirely, because the saved state belongs to the model that produced it. If you want a cheaper model for a sub-task, spawn a separate call for it rather than swapping the model on your main loop, and keep the main loop on one model so its cache survives. Caching pairs naturally with routing calls to the cheapest capable model: route to keep the per-call price low, cache to keep the repeated prefix nearly free, and you attack the bill from both directions.

The takeaway

An agent is a machine that resends almost the same enormous prompt over and over. That is a liability if you pay full price for it every step, and an asset the moment the provider is reusing its own prefill state instead. The whole game is arranging the prompt so the stable part comes first and never changes, then confirming with the usage numbers that it is actually being reused.

Freeze the system prompt and the tool list. Push everything volatile, timestamps, IDs, the current question, to the end, after your breakpoint. Treat the conversation history as append-only and never edit what you already sent. Then read cache_read_input_tokens on every response and make sure it is doing the work. That is a few lines of placement discipline, and on a busy agent it is the difference between a bill you can defend and one you have to explain.

If your agents are running up a token bill that feels out of proportion to what they actually do, this is exactly the kind of production cost work I help teams get right. Book a consultation call and we can look at where your prompts are paying full price for tokens the provider already processed.