Your Easy Queries Are Paying for Thinking They Never Use

You turned on extended thinking a few months ago because the quality numbers went up and the demo felt smarter. Nobody argued. Then the finance review came around and the model line had roughly doubled, on flat traffic, and someone asked the reasonable question of what changed. The honest answer is that you started paying for every request to think hard, including the overwhelming majority that had nothing to think about.

This is the part of reasoning models that the launch posts skip. The thinking tokens are real tokens, billed at the output rate, and on a reasoning model they are usually the single largest and most variable slice of the bill. A request that used to be a couple hundred output tokens can quietly become a few thousand once the model decides to reason its way to the same answer it would have given instantly. Turn that on across the board and you have converted a predictable cost into one that moves with how chatty the model feels, which is not a line item finance enjoys.

The fix is not to turn thinking off. It is to stop treating it as one global switch. Reasoning effort is a per-query decision, and the leverage comes from admitting that your traffic is not uniform.

The distribution is the whole argument

Pull a day of real requests and label them by how much reasoning they actually need. Almost everyone who does this finds the same shape: a long tail of genuinely hard queries that justify every thinking token, sitting on top of a large body of easy ones. Classification, a lookup phrased as a question, a short rewrite, a yes or no gate, a greeting. None of those need a chain of thought, and running one on them buys you nothing but latency and spend.

Put a number on what uniform effort is costing you. This is deliberately simple arithmetic, because the point is to see the waste, not to model it perfectly.

# Rough cost of a policy that thinks hard on every request, versus one
# that only thinks hard on the fraction that needs it.
OUTPUT_PRICE_PER_TOKEN = 25 / 1_000_000  # $25 / 1M output tokens, a frontier tier

def daily_thinking_spend(requests_per_day, hard_fraction,
                         think_tokens_hard=3000, think_tokens_easy=200):
    """Compare a think-everything policy against a matched-effort policy."""
    think_all = requests_per_day * think_tokens_hard
    matched = requests_per_day * (
        hard_fraction * think_tokens_hard
        + (1 - hard_fraction) * think_tokens_easy
    )
    return {
        "think_all_usd": think_all * OUTPUT_PRICE_PER_TOKEN,
        "matched_usd": matched * OUTPUT_PRICE_PER_TOKEN,
        "saved_pct": 100 * (1 - matched / think_all),
    }

# 200k requests/day, only 20% of them genuinely hard:
print(daily_thinking_spend(200_000, hard_fraction=0.20))
# -> {'think_all_usd': 15000.0, 'matched_usd': 3800.0, 'saved_pct': 74.6...}

Three quarters of the thinking spend on that traffic is going to requests that did not need it. Your real numbers will differ, but the structure holds wherever the easy queries outnumber the hard ones, which is almost everywhere outside a pure research workload. That gap is the budget you are trying to recover, and you recover it by matching effort to difficulty one request at a time.

Classify difficulty before you spend on it

The mechanism is a cheap gate in front of the expensive call. Something fast and small looks at the request, decides how hard it is, and picks an effort tier. The gate has to be cheap relative to what it saves, so it is either a heuristic or a small non-reasoning model, never the reasoning model itself.

from enum import Enum

class Effort(str, Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"

# A heuristic first pass catches the obvious cases for free. Only the
# ambiguous middle needs a model to weigh in.
def quick_effort(prompt: str) -> Effort | None:
    text = prompt.strip()
    if len(text) < 200 and "?" not in text:
        return Effort.LOW          # short, declarative, likely trivial
    hard_signals = ("step by step", "prove", "reconcile", "debug",
                    "trade-off", "given the constraints")
    if any(s in text.lower() for s in hard_signals):
        return Effort.HIGH
    return None                    # unsure, ask the small classifier

CLASSIFIER_PROMPT = """Rate how much step-by-step reasoning this task needs.
Reply with exactly one word: low, medium, or high.
low = direct recall, formatting, or a one-step answer.
medium = a couple of dependent steps.
high = multi-step reasoning, planning, or careful trade-offs.

Task: {task}"""

async def classify_effort(small_model, prompt: str) -> Effort:
    fast = quick_effort(prompt)
    if fast is not None:
        return fast
    # A small, non-reasoning model. Cap output so this stays a few cents
    # per thousand calls and a few tens of milliseconds.
    reply = await small_model.complete(
        CLASSIFIER_PROMPT.format(task=prompt[:2000]), max_tokens=1
    )
    return Effort(reply.strip().lower()) if reply.strip().lower() in Effort._value2member_map_ else Effort.MEDIUM

The heuristic layer matters more than it looks. A large share of production traffic is templated or short enough that you can tier it without a model call at all, which keeps the classifier off the critical path for the easy cases that are the whole point. Reserve the model call for the genuinely ambiguous middle.

Turn the tier into the provider's actual knob

Once you have a tier, you set it on the request. The control differs by provider, and it is worth being precise here because the old mental model is stale. On current Claude models, thinking is adaptive and you steer depth with an effort setting that runs from low through max; the fixed budget_tokens field that people reach for out of habit is rejected on the newer models. Gemini exposes a thinking budget parameter, where zero disables thinking and a value of negative one hands the sizing back to the model. Same idea, different dials.

async def answer(prompt: str, *, small_model, reasoning_model) -> str:
    tier = await classify_effort(small_model, prompt)

    # Map your internal tier onto whatever the model actually accepts.
    # Claude: adaptive thinking + an effort level (low..max); no budget_tokens.
    result = await reasoning_model.create(
        model="claude-opus-5",
        max_tokens=4096,
        thinking={"type": "adaptive"},
        output_config={"effort": tier.value},   # "low" | "medium" | "high"
        messages=[{"role": "user", "content": prompt}],
    )
    # Gemini equivalent would pass thinking_budget: 0 for low,
    # a small cap for medium, and -1 (model-sized) for high.
    return result.text

That is the entire happy path. A cheap classifier, a tier, and the provider knob. Most of the savings from the earlier arithmetic land right here, because you have stopped sending the easy majority down the deep path.

Put a ceiling on the whole thing

Matching effort to difficulty lowers the average. It does not cap the total, and a bad day, a traffic spike, or a tenant hammering hard queries can still run the bill past what you planned. So the last piece is a spend envelope that the classifier answers to. When the day's thinking budget is running low, you bias the tiers down: the hard queries still get their reasoning, but the borderline ones get demoted rather than indulged.

import time

class ThinkingBudget:
    """A soft daily ceiling on thinking spend that degrades gracefully."""
    def __init__(self, daily_token_cap: int):
        self.cap = daily_token_cap
        self.spent = 0
        self.day = time.gmtime().tm_yday

    def _roll(self):
        today = time.gmtime().tm_yday
        if today != self.day:
            self.spent, self.day = 0, today

    def record(self, thinking_tokens: int):
        self._roll()
        self.spent += thinking_tokens

    def adjust(self, tier: Effort) -> Effort:
        self._roll()
        used = self.spent / self.cap
        if used < 0.8:
            return tier                      # plenty of budget, honor the tier
        if used < 1.0 and tier is Effort.MEDIUM:
            return Effort.LOW                # tighten the borderline cases
        if used >= 1.0 and tier is not Effort.HIGH:
            return Effort.LOW                # over budget: only true-hard thinks
        return tier                          # never starve a HIGH query

The design choice worth calling out is that HIGH is never demoted. The point of the budget is to shed the spending you would not have missed, not to sabotage the requests that justified buying a reasoning model. Feed the actual thinking token count from each response back into record, and this stays honest across the day. It is the same posture as a proper per-tenant spend ledger, scoped to the one cost that reasoning models made volatile.

Where this bites you

A few things go wrong in practice, and they are all avoidable if you know to look.

The classifier can cost more than it saves if you are careless. If you route the difficulty decision to a slow or expensive model, you have just added a second reasoning call in front of the first. Keep it small, cap its output to a token or two, and lean on the heuristic layer so most requests never reach it.

Averages will hide a regression. If you drop effort and watch a single blended quality score, you can degrade the hard 20 percent badly while the easy 80 percent props the average up. Measure accuracy per difficulty tier, not as one number, and wire it into your eval suite so a regression fails in CI instead of showing up in a support queue. The whole scheme rests on the classifier being roughly right, so the classifier itself is something you evaluate, not something you trust.

Adaptive models change the shape underneath you. On models that decide their own thinking depth, your effort setting is an upper bound and a nudge, not a fixed dial, so two requests at the same tier can cost different amounts. That is fine for budgeting in aggregate, but it means you should track the thinking tokens you actually spent from each response rather than assume the tier fixed them. That accounting is the same discipline as handling reasoning traces at the boundary instead of trusting whatever the model streams back.

And do not confuse this with model routing. Routing changes which model answers; effort budgeting changes how hard one model thinks. They stack, but reach for effort first, because it keeps a single cache namespace and one behavior profile, and on current models a low effort setting on the strong model often beats a previous generation model at full effort. Prove out the simple version before you build the multi-model cascade.

The takeaway

Extended thinking is not expensive because it is a bad feature. It is expensive because most teams buy it in bulk for traffic that is mostly easy. Classify each request, spend deep reasoning only where the difficulty earns it, and hold the total under a ceiling that degrades the borderline cases before it ever touches the hard ones. You keep the quality on the requests that made you want a reasoning model, and you hand finance back a line item that no longer moves with the model's mood.

If your reasoning bill jumped and you are not sure how much of it is real work versus reflexive overthinking, that is usually a quick thing to measure and a quicker one to fix. Book a consultation call and we can find the easy majority you are overpaying for.