You Turned Off Thinking and Your Agent Stopped Calling Tools

A team I was helping moved one step of their pipeline, a document extractor that pulled fields and occasionally called a lookup tool to resolve a vendor ID, onto a smaller, faster model to cut cost. They did the sensible thing and turned the thinking knob all the way down, because the step felt mechanical and they wanted it quick and cheap. The extraction numbers held. What did not hold was the vendor resolution: a slow drip of records came back with a plausible vendor ID that no lookup had produced. The schema was satisfied every time. Nothing errored. The tool had simply stopped being called, and the model had learned to guess a value that fit the shape.

The knob you turned down is not the knob you think it is

On the previous generation of small models, the way you asked for less thinking was a token budget: you set a number, the model spent up to that many tokens reasoning, and if you set it to zero you got a fast, cheap, non reasoning call. That mental model is now wrong. On the newest small models the budget field is gone and the control is an effort setting, and the two are not the same thing. Anthropic's what is new in Claude Haiku 5.5 lists the old budget_tokens field as a breaking change: it returns an error, and you are told to replace it with adaptive thinking plus an effort level.

The important part is what effort touches. The effort parameter documentation is explicit that effort affects all tokens in the response, including the tool calls and the function arguments, not only the hidden reasoning. Lower effort means fewer and terser tool calls. It makes the model proceed directly to action without preamble and combine several operations into one call. Read that again with a production hat on: the setting you reach for to save money is the same setting that governs whether your agent reaches for its tools at all. It is a behavioral signal, not a strict budget, so two requests at the same level can even behave differently.

This is why the old habit fails. You are not turning down a cost meter that sits off to the side. You are turning down the model's willingness to do the work.

Why turning it down breaks tool use

Calling a tool is a decision, and decisions take a little deliberation. The model has to notice that the answer is not already in its context, judge that a tool would produce it, and choose the call over just answering. That noticing is cheap but it is not free, and it is exactly the kind of step that thinking buys you. Strip the thinking out and you bias the model toward the fastest path to a finished looking response, which is to answer directly.

Now add a constrained output format on top, which is the normal setup for an extractor or any step that feeds code. The Haiku 5.5 prompting guide spells out the trap in one line: with thinking off, the model might skip a tool call it needs when you also request JSON output with structured outputs. The schema is a strong pull toward emitting a shaped object immediately, and with no deliberation step in the way, the model satisfies the schema with its best guess rather than pausing to fetch the real value. That is the precise failure the team hit.

It is worth being clear that this is not one vendor's quirk. The pattern generalizes to any model with an effort or thinking control: the deliberation you are paying for is partly the deliberation that chooses to use tools, so cutting it degrades tool use first. The same forces show up in plain tool call reliability, where validation and recovery exist precisely because the decision to call and the arguments passed are both fallible. Lower effort widens that fault.

The failure is silent and schema shaped

The cruel part is how quiet it is. If a skipped tool call produced malformed output, you would catch it at the boundary and alert. Instead it produces well formed output. The guessed vendor ID is a string where a string belongs. It passes JSON parsing. It passes schema validation. Everything downstream that trusts the contract accepts it, and the record flows into the system looking exactly like a record that did the real lookup.

This is where a structured output defense gives you a false sense of safety. A validation gateway is the right pattern, and if you have read why almost valid JSON is quietly breaking you you already route every structured call through one schema and two gates. But a gateway checks shape and types, not provenance. It cannot tell a resolved vendor ID from a hallucinated one, because both are valid strings. The schema was never the thing protecting you from a skipped tool call. You need the tool to have actually run.

So the detectable surface is small. The only honest signals are at the level of behavior: did a tool call appear in the trace when one should have, and does the returned value match what the tool would have produced. Neither of those is visible to a schema validator, which is why a route that looks healthy on parse rate can be rotten on correctness.

Measure the capability, not the token count

Because effort trades capability for cost, you cannot choose a level by looking at the price list. You have to measure what each level does to the behavior you care about, on your own traffic. The tool for this is an effort sweep: take a frozen set of labelled cases for one route, run every effort level against it, and record the behavioral metrics per level rather than a single blended score.

# Sweep effort levels on one route and record behavioral metrics.
# The point is to see where tool-calling and schema fidelity fall off,
# not to find the cheapest level that still "runs".
import statistics

LEVELS = ["off", "low", "medium", "high"]  # "off" = thinking disabled

def run_route(case, level):
    # thinking disabled only when level == "off"; else adaptive thinking + effort
    resp = call_model(case.input, level=level, tools=TOOLS, schema=INVOICE)
    return {
        "tool_called": any(b.type == "tool_use" for b in resp.content),
        "schema_valid": validate(resp, INVOICE),
        "correct": resp_value(resp) == case.expected,  # graded against truth
    }

def sweep(cases):
    for level in LEVELS:
        rows = [run_route(c, level) for c in cases]
        recall = statistics.mean(r["tool_called"] for r in rows if case_needs_tool(r))
        valid = statistics.mean(r["schema_valid"] for r in rows)
        acc = statistics.mean(r["correct"] for r in rows)
        print(level, round(recall, 3), round(valid, 3), round(acc, 3))

Reading the output is the whole exercise. Schema validity will often look flat across levels, which is the trap: it stays high even as the answers go wrong. Tool call recall and end to end accuracy are the columns that tell the truth, and the level where either of them drops sharply is the floor for that route. The discipline is the same one behind matching reasoning effort to query difficulty: a single quality number hides the regression, so you measure per behavior and let the cliff show itself. The difference is that here the cliff is tool use and provenance, not just answer quality on hard prompts.

Set effort per route, not as a global cost lever

Once you can see the floor, the fix is to stop treating effort as one dial on the whole system and start treating it as a per route capability setting that an eval gate signs off on. A chat route with no tools and no schema can run at the cheapest level, or with thinking disabled, and lose nothing. A tool using or structured route holds adaptive thinking on and sets the lowest effort level that stayed above its floor in the sweep.

# Effort is chosen per route from the sweep, and tool routes keep thinking on.
ROUTE_POLICY = {
    "support_chat":     {"effort": "low",    "thinking": "disabled"},
    "label_classify":   {"effort": "low",    "thinking": "disabled"},
    "invoice_extract":  {"effort": "medium", "thinking": "adaptive"},  # calls lookup tool
    "agent_planner":    {"effort": "high",   "thinking": "adaptive"},  # long tool loop
}

def request_for(route, payload):
    p = ROUTE_POLICY[route]
    req = {"input": payload, "output_config": {"effort": p["effort"]}}
    if p["thinking"] == "adaptive":
        pass                       # omit the thinking field: adaptive is the default
    else:
        req["thinking"] = {"type": "disabled"}
        # on a route that must call a tool, drop the strict JSON format instead,
        # or force the call, so a skipped tool cannot hide behind a valid schema
    return req

The prompting guide offers three concrete escapes when you do need thinking off on a route that still calls tools: leave adaptive thinking on for those requests, remove the strict output format from any request where the model must call a tool, or force the call with a tool choice setting. In practice the first option is the one to default to, and you reserve thinking disabled for the genuinely tool free routes. This is a different axis from routing between a cheap and an expensive model: routing changes which model answers, effort changes how hard one model works, and they stack. Reach for the right effort per route before you build a cascade.

The parts that bite

A few things will catch you even after you accept that effort is a capability dial.

Changing the top level effort between requests invalidates the prompt cache for that conversation. If you were counting on cache hits across a long session to keep prefix caching cheap, varying effort mid stream quietly throws that saving away. Pick a level per route and hold it constant within a cached conversation, or use the per message effort change where the provider supports it, which preserves the cache.

Thinking tokens now count toward your output limit. A max_tokens value you sized for a non thinking small model can stop the response after the reasoning and before any text, which looks like an empty or truncated reply rather than a configuration mistake. Leave headroom when adaptive thinking is on.

Low effort does not only skip tools. The same documentation notes it also makes the model more likely to skip a search it should run, stop early in a long agent prompt and hand the task back, or report a code change as done without checking it. If you dropped effort to save money on an agent, these are the symptoms to watch for, and they are easy to blame on the model when the setting is the cause. Migrating to a new small model is the moment to rerun this whole analysis under an eval gate before you cut over, because the floors move between model versions.

Finally, remember the metric trap from the sweep. If your dashboards watch parse rate and latency, a route degraded by low effort will look perfectly healthy while it ships wrong data. The signal lives in tool call recall and graded correctness, so put those in your CI eval suite and alert on them, not on the shape of the output.

The takeaway

The small models got a new knob, and the knob is not the one the old habit expects. Effort is not a cost meter you can turn to zero without consequence. It shapes every token the model produces, including the decision to call a tool and the care taken to fill a schema, so turning it down or off is the same motion as asking the model to do less of the work you deployed it for. The damage lands first and quietest on tool use and structured output, where a skipped call hides inside a valid looking object and sails past every gate you built to catch bad shapes.

Treat effort as a per route capability setting, not a global cost lever. Measure the floor for each route with an effort sweep, keep adaptive thinking on wherever a tool must run, and hold the level constant inside a cached conversation. If you moved a step onto a cheaper model or a lower effort setting and your tool calls or JSON got quietly worse, that gap is usually fast to find and faster to fix. Book a consultation call and we can measure what your cost change actually cost you.