Your Token Budget Is Wrong the Day You Switch Models

A team I was helping moved a classification agent from one small model to its newer version over a weekend. Same prompts, same code, one string changed in the config. On Monday the spend graph was up by about a third, and a few requests had started failing with a context overflow error that had never fired in six months of running. Nobody had edited a prompt. The new model was cheaper per token on the price page. The bill went the other way.

The cause was not the prompts and not a bug in their code. It was the tokenizer. The newer model counts the same text as more tokens, and every number the team had measured against the old model, cost per request, headroom under the context limit, the point where a prompt is eligible for caching, was quietly measured in the wrong unit the moment they flipped the config. None of it threw until a request landed just over a limit.

This is one of the easiest production surprises to avoid and one of the most common to hit, because the thing that changed is invisible in a diff. A model id is one line. The tokenizer behind it is not in your repository at all.

The tokenizer ships with the model

A language model does not read characters. It reads tokens, and a tokenizer is the function that turns your bytes into the integer sequence the model actually sees. That function is part of the model, not a neutral property of your text. Different model families use different tokenizers, and even within one vendor a new generation can introduce a new one.

OpenAI makes this concrete in the tiktoken library, where encoding_for_model maps a model name to an encoding and newer models use o200k_base while older ones use cl100k_base. The same string run through the two encodings produces two different token counts. Anthropic documents the same effect across its own line: the token counting guide notes that Claude 4.7 and later models use a newer tokenizer under which the same input text produces roughly thirty percent more tokens than on earlier models, with the exact increase depending on the content.

That is the whole mechanism. Your text is fixed. The number of tokens it becomes is a property of the model you send it to. So "how many tokens is this prompt" has no answer until you also say which model, and the answer you cached from the last model is no longer the answer for the next one.

The latest small model release makes the gap hard to ignore. Anthropic's notes for Claude Haiku 5.5 list "same text counts as more tokens" as a behavior change with a plain action attached: recount prompts and revisit max_tokens and cost estimates. The page is telling you the budget moved before you measured it.

Four things that break without throwing

The reason this bites is that the token count is not one number used in one place. It is an input to four separate systems, and all four shift together while every one of them keeps running.

Cost is the obvious one. Your estimate was price per token multiplied by a token figure. The new model changes both terms. A lower rate times more tokens can net out higher, flat, or lower depending on your content, and you cannot know which without recounting. Worse, the higher count can push a prompt across a long context pricing boundary, the point past which every token on that request is billed at an elevated rate. A prompt that sat comfortably under that boundary on the old model can cross it after the recount, moving a whole class of requests into the expensive band at once.

Headroom under max_tokens is the second. If you set an output limit assuming an input of a certain size, a larger input eats into the context window you were counting on. On models where thinking tokens count toward the output limit, a small max_tokens can now stop a response after an internal thinking block and before any text reaches you. This is where the overflow errors come from, and it is worth reading alongside how to budget reasoning effort to the difficulty of the query so the thinking does not silently consume the room you meant for the answer.

Context window fit is the third, and it is the same arithmetic from the other side. A retrieval step that packed documents up to a token target now overshoots, because each document counts for more. Teams that moved to a large window and stopped retrieving are especially exposed, which is why it pays to spend the window with a token budget rather than filling it to a number that no longer means what it did.

Prompt caching is the fourth and the most quietly expensive. A cache key is computed over the tokenized prefix. Change the tokenizer and none of your existing entries match, so every prefix is paid in full again on the first call, and your hit rate falls off a cliff on migration day before it climbs back. Minimum cacheable lengths can change too: Haiku 5.5 lowered its minimum from 4,096 tokens to 512, which changes which of your prompts are even eligible. If you rely on cached prefixes to hold cost and latency down, a model swap is a cold cache, and planning for that is cheaper than discovering it.

Recount on the model you are moving to

The fix is not clever. It is to stop reusing counts measured against the old model and take fresh ones against the new model before the migration, not after. Most providers expose a token counting endpoint that counts under the tokenizer of whatever model id you pass, which is exactly the lever you need: count the same request twice, once with each model, and read the difference off the two numbers.

import anthropic

client = anthropic.Anthropic()

def count(model, messages, system=None):
    # counts under the tokenizer of THIS model id, not a fixed one
    r = client.messages.count_tokens(model=model, system=system, messages=messages)
    return r.input_tokens

def migration_delta(old_model, new_model, sample):
    # sample is a list of real requests, not one toy prompt
    old = sum(count(old_model, m["messages"], m.get("system")) for m in sample)
    new = sum(count(new_model, m["messages"], m.get("system")) for m in sample)
    return old, new, (new - old) / old

old, new, ratio = migration_delta("claude-haiku-4-5", "claude-haiku-5-5", sample)
print(f"{old} -> {new} tokens, {ratio:.0%} change on your traffic")

Run it across a representative slice of real prompts, not one example, because the increase depends on the content: prose, code, and structured data move by different amounts. The output is a single ratio you can multiply through every cost estimate and every headroom check you own. Anthropic's guidance is explicit that you should count your prompts with the model id you plan to use rather than reuse a figure from an older model, and the endpoint is free to call.

Put the count where the budget lives

A one off script answers the question once. The deeper fix is to stop treating "tokens in this prompt" as a constant in your code and start treating it as a value that depends on the active model. Anywhere you have a hardcoded token figure, a context limit you compare against, an assumed input size for a cost line, a cache eligibility threshold, is a place that will silently drift the next time the model id changes.

Keep the model facts in one record and read from it, so a migration is a data change in one place rather than a hunt through the codebase.

# models.yaml: the numbers that move when the model moves
claude-haiku-4-5:
  context_window: 200000
  long_context_boundary: null     # flat pricing on this model
  min_cacheable_tokens: 4096

claude-haiku-5-5:
  context_window: 1000000
  long_context_boundary: 100000   # tokens above this bill at the higher band
  min_cacheable_tokens: 512       # shorter prompts are now cacheable

Now the budget logic reads the record for the active model and recounts against it, instead of comparing a live request to a number that was true for a model you no longer run.

import yaml

MODELS = yaml.safe_load(open("models.yaml"))

def check_request(model, messages, max_tokens, system=None):
    spec = MODELS[model]
    tokens = count(model, messages, system)           # recount, never reuse
    if tokens + max_tokens > spec["context_window"]:
        raise ValueError(f"{tokens}+{max_tokens} over {model} window")
    band = spec["long_context_boundary"]
    tier = "long" if band and tokens > band else "standard"
    cacheable = tokens >= spec["min_cacheable_tokens"]
    return {"tokens": tokens, "price_tier": tier, "cacheable": cacheable}

This is the same discipline that makes a model swap survivable in the first place: name the model by role, keep its properties in one place, and let a migration be a value you change rather than a set of assumptions scattered across the code. It sits naturally next to the habit of treating a model upgrade as the dependency change it actually is rather than a string bump.

Gate the migration on the recount

Once the count is a function of the model, you can put a gate in front of the switch so a surprise bill cannot ship. The gate is small: recount a fixed sample on both models and fail if the change exceeds what you budgeted for, or if any request crosses the context window or a pricing boundary it did not cross before.

Wire this into the same release check that holds your other migration risks, so the token delta is reviewed next to the eval results and not discovered on the invoice. If you run per customer spend caps, feed the recounted figure into them too, because per-tenant budgets computed on the old tokenizer will under count every tenant the day you migrate. The point is not to block the move. It is to make the cost and the overflow risk a number someone approved, the way you would approve any other dependency bump.

The parts that bite

A token counting endpoint returns an estimate, not the exact figure you will be billed. Providers note the real count can differ by a small amount, and some add system tokens you are not charged for. For migration planning this is fine, because you care about the ratio between two models and the proximity to a limit, not the last few tokens. Do not build a check that fails on a one token difference.

Multi turn conversations can grow faster than the tokenizer change alone explains. Some newer models keep the thinking blocks from every earlier assistant turn in context and count them as input, where the previous generation kept only the latest turn. Haiku 5.5 does exactly this. So a long session accumulates input tokens from two causes at once, the tokenizer and the retained thinking, and a context budget that looked safe on turn one can overflow deep in a conversation. This is a context management problem as much as a cost one, and it is worth pruning retained blocks the way you would prune any stale context from a long running agent.

Routing across providers multiplies the problem. If you route requests across model families to cut cost, you are not reasoning about one tokenizer, you are reasoning about several, and a prompt that fits one model's window and price tier may not fit another's. The budget is per model, so the recount is per model, and the router has to carry those facts rather than assume a shared count.

Finally, images and documents count under the same model specific rules, and the endpoint will count them if you send them as base64. A vision heavy prompt can shift by a different amount than a text one, so sample the real mix of modalities your traffic sends rather than a page of prose.

The takeaway

The number of tokens in a prompt feels like a property of the prompt, so it is easy to measure it once and treat it as settled. It is not. The tokenizer ships inside the model, and a migration swaps it out underneath four systems at once: your cost estimate, your output headroom, your context fit, and your cache. Nothing in the diff shows it, and nothing throws until a request lands just past a limit, which is why the surprise always arrives as a bill or an overflow rather than a failed test.

The work to avoid it is an afternoon. Recount a real sample on the model you are moving to, put the counts and the model facts in one place your budget logic reads, and gate the switch on the delta so the cost is a decision and not a discovery. If a recent model swap moved your spend or your cache hit rate while your prompts stayed exactly the same, that gap is almost always the tokenizer, and it is worth a short look together before the next invoice closes.