LLM Model Routing: Cut AI Costs Without Losing Quality

Here is a bill nobody wants to explain to their CFO. You wired your product to your best model because it gave the cleanest demo, shipped it, and now every single request, from "reset my password" to "rewrite this legal clause," runs through the most expensive model you have access to. The password resets work great. They also cost the same per token as the hard stuff. You are paying flagship prices for work a model costing a fifth as much would nail every time.

This is one of the most common ways AI products quietly bleed money, and it got sharper this month. Anthropic shipped Claude Sonnet 5, which lands close to flagship Opus quality on a lot of tasks while costing roughly half as much, and it is running introductory pricing on top of that. When the gap between "good enough" and "the best" narrows and the price gap stays wide, routing stops being a nice-to-have and becomes the obvious lever. This post covers why the money is bigger than most people expect, a router you can drop in today, a cascade for when correctness is checkable, and the traps that make naive routing worse than no routing at all.

The business problem: you are overpaying on the easy majority

Real traffic is lopsided. Whatever your product does, the request distribution almost always has a long, fat head of easy work and a thin tail of genuinely hard work. Support bots spend most of their day on FAQ-shaped questions. Coding assistants spend most of theirs on small edits, not architecture. Classification and extraction pipelines are easy by definition. The hard tail is real and it matters, but it is the minority.

Now put prices next to that. On the Claude lineup, Haiku 4.5 runs $1 per million input tokens, Sonnet 5 runs $3 (with introductory pricing lower through the end of August), and Opus 4.8 runs $5. Output tokens follow the same shape, wider apart. So if you route everything to Opus, every trivial request pays a five-times premium over Haiku for output it did not need.

Do the arithmetic on a realistic mix. Say you handle a million requests a month, seventy percent of them simple, twenty percent moderate, ten percent hard. Send it all to Opus and you pay the Opus rate on all million. Route the simple seventy percent to Haiku and the moderate twenty percent to Sonnet, and you have moved ninety percent of your volume onto models that cost a fraction of the flagship, while the hard ten percent still gets the best model you have. The quality on the hard requests is identical, because they still go to the same model they always did. You just stopped overpaying on everything else.

That is the whole idea. Routing is not about accepting worse answers to save money. It is about matching each request to the cheapest model that can actually handle it, so you only pay flagship prices for work that actually needs flagship intelligence.

Pattern one: a predictive router

The cleanest version of routing is a classifier that reads the incoming request, guesses how hard it is, and picks a model before you spend anything on the real answer. You use a small, cheap model as the classifier, so the routing decision itself costs almost nothing.

Start with the registry. Keep it boring and explicit, so anyone on the team can see what routes where.

from anthropic import Anthropic

client = Anthropic()

# One place that maps a difficulty tier to a concrete model.
# Change a route here, not scattered through your codebase.
TIER_TO_MODEL = {
    "simple": "claude-haiku-4-5",    # cheap, fast, great at scoped tasks
    "moderate": "claude-sonnet-5",   # near-flagship quality, ~half the cost
    "hard": "claude-opus-4-8",       # reach for this only when you need it
}

Now the classifier. A cheap model, a tight system prompt, and structured output so you get back a clean enum instead of a paragraph you have to parse. Structured outputs constrain the response to your schema, which is exactly what you want for a routing decision.

import json

# Force the classifier to answer with one of three tiers, nothing else.
ROUTING_SCHEMA = {
    "type": "object",
    "properties": {
        "tier": {"type": "string", "enum": ["simple", "moderate", "hard"]},
    },
    "required": ["tier"],
    "additionalProperties": False,
}

CLASSIFIER_SYSTEM = (
    "You are a routing classifier. Read the user's request and rate how much "
    "model capability it needs.\n"
    "- simple: lookups, short rewrites, formatting, single-step answers.\n"
    "- moderate: multi-step reasoning, code with real logic, careful analysis.\n"
    "- hard: novel problem-solving, long-horizon planning, subtle correctness.\n"
    "Rate the difficulty of the task, not how long the answer is."
)

def classify(prompt: str) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=64,
        system=CLASSIFIER_SYSTEM,
        messages=[{"role": "user", "content": prompt}],
        output_config={"format": {"type": "json_schema", "schema": ROUTING_SCHEMA}},
    )
    text = "".join(b.text for b in resp.content if b.type == "text")
    return json.loads(text)["tier"]

Wire it together and you have a router. Classify first, then send the real request to whatever model that tier points at.

def route_and_answer(prompt: str) -> tuple[str, str]:
    tier = classify(prompt)
    model = TIER_TO_MODEL[tier]

    resp = client.messages.create(
        model=model,
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}],
    )
    answer = "".join(b.text for b in resp.content if b.type == "text")
    return model, answer  # return the model so you can log what got picked

The classifier call adds a little latency and a little cost, but a Haiku classification on a short prompt is cheap enough to disappear against the savings on the real request. The one rule that trips people up: never let the classifier get more expensive than the work it is routing. If your classifier prompt starts ballooning with examples and instructions, you have reinvented the problem you were trying to solve.

Pattern two: a cascade that escalates on failure

Prediction works well when difficulty is visible in the request. Plenty of the time it is not. Two requests can look identical and one turns out trivial while the other is a trap. For those cases, flip the logic: try the cheap model first, check whether the answer is actually good, and escalate only if it is not. This is a cascade, and it works whenever correctness is checkable after the fact.

The core loop walks from cheapest to most expensive and stops at the first model whose answer passes a check.

CASCADE = ["claude-haiku-4-5", "claude-sonnet-5", "claude-opus-4-8"]

def generate(model: str, prompt: str) -> str:
    resp = client.messages.create(
        model=model,
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(b.text for b in resp.content if b.type == "text")

def answer_with_cascade(prompt: str) -> tuple[str, str]:
    reply = ""
    for model in CASCADE:
        reply = generate(model, prompt)
        if is_good_enough(prompt, reply):
            return model, reply
    # Nothing passed; return the strongest model's attempt anyway.
    return CASCADE[-1], reply

The interesting part is the check. Do not ask the model that wrote the answer whether its own answer is good, because self-reported confidence is close to useless. Use a separate, cheap grader with a clear bar. Structured output again, so you get a clean boolean back.

GRADE_SCHEMA = {
    "type": "object",
    "properties": {"adequate": {"type": "boolean"}},
    "required": ["adequate"],
    "additionalProperties": False,
}

def is_good_enough(prompt: str, reply: str) -> bool:
    # A small model grading against an explicit rubric, not vibes.
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=32,
        system=(
            "You grade whether an answer fully and correctly satisfies the "
            "request. Return adequate=false if it is wrong, incomplete, evasive, "
            "or misses a constraint in the request. Be strict."
        ),
        messages=[{
            "role": "user",
            "content": f"Request:\n{prompt}\n\nAnswer:\n{reply}",
        }],
        output_config={"format": {"type": "json_schema", "schema": GRADE_SCHEMA}},
    )
    text = "".join(b.text for b in resp.content if b.type == "text")
    return json.loads(text)["adequate"]

When the grader is cheaper and more reliable than the generation it guards, the cascade pays for itself: most requests clear the first rung, and only the ones that genuinely need more horsepower ever touch the expensive model. If you can replace the model grader with a deterministic check, do it. Code that compiles, JSON that parses against a schema, a SQL query that runs without error, an extraction that contains the required fields. A real verifier beats a model verifier every time it is available, on both cost and trust.

The traps that make routing backfire

Routing done badly is worse than not routing, because now you have two systems that can fail instead of one. Watch for these.

• The router costs more than it saves. Every classifier call and every grader call is real spend. If your traffic is almost all hard requests, or your prompts are tiny, the routing overhead can swallow the savings. Measure your actual distribution before committing. Routing pays off when the easy majority is large, and not before.
• Misrouting the hard tail is expensive in the other currency. Sending a genuinely hard request to Haiku and getting a confident, wrong answer can cost you a customer, which dwarfs the token savings. Bias your thresholds so the failure mode is "escalated a request that did not need it," not "answered a hard request with a weak model." Overpay a little rather than underdeliver.
• Latency stacks up in a cascade. A request that fails Haiku, fails Sonnet, then finally lands on Opus has paid three round trips of latency. For interactive UI that can feel broken. Cap the cascade depth, and for latency-sensitive paths lean on the predictive router instead so you commit to one model up front.
• You cannot tune what you do not log. Record the chosen model, the classifier's tier, and, where you can measure it, whether the answer held up. Without that you are flying blind. The whole system is a set of thresholds, and thresholds only get better when you can see how the current ones perform.

That last point deserves a line of code, because it is the difference between a router you can improve and one you just hope is working.

def route_logged(prompt: str) -> str:
    model, answer = route_and_answer(prompt)
    log_routing_event(
        model=model,
        prompt_len=len(prompt),
        # attach a downstream success signal when you have one:
        # thumbs-up, no retry, ticket resolved, tests passed.
    )
    return answer

Feed that log back into your tier definitions every so often. If "moderate" requests keep getting escalated by the cascade, your classifier is underrating them and the threshold needs to move. Routing is not a set-and-forget config. It is a dial you keep turning as your traffic and the model lineup change, and the lineup changes often.

The takeaway

One model for everything is simple, and simple is worth something, but past a certain volume it is just a way to overpay on the easy majority of your traffic. Routing fixes that without touching the quality of your hard requests, because those still go to your best model. A predictive router works when difficulty is visible up front. A cascade works when correctness is checkable after the fact. Most mature setups use both, and all of them live or die on logging and honest thresholds.

The reason this is worth doing right now is that the models moved. When a mid-tier model gets close to flagship quality at half the price, the cheapest capable model for a huge share of your traffic just got a lot more capable, and the savings from routing to it got a lot bigger. If you are still paying flagship rates on your whole pipeline, this is where the low-hanging fruit is. For a related angle on keeping long-running agents cheap, see the piece on context compaction for long-running AI agents.

If you want a second set of eyes on your model mix, or you are staring at an AI bill that grew faster than your usage did, book a consultation call and we will map your traffic to the right models together.