Your Agent Has 200 Tools and Picks the Wrong One

An agent with eight tools is a joy to work with. It reads the request, picks the obvious tool, and gets on with it. Then the integrations pile up. You connect the CRM, the ticketing system, the data warehouse, three internal services, and a couple of MCP servers someone stood up last quarter, and now the same agent is carrying two hundred tool definitions into every single turn. It pays for all of them on every request, before the user has finished typing, and it still fumbles the choice, because you have asked a model to pick one right tool out of a wall of near-identical schemas.

This is the tax nobody plans for. Each integration looks cheap in isolation. A tool definition is a name, a description, and a parameter schema, a few hundred tokens at most. The cost is that they compound, and they compound in two directions at once. The bill goes up, because every schema rides along on every turn whether the request needs it or not. And the quality goes down, because selecting from two hundred options, forty of which are variations on search and update and list across different services, is genuinely hard, and models get worse at it as the list grows. You end up paying more to get a worse answer.

The two costs of a fat toolbox

Start with the token cost, because it is the easy one to see. If your tool definitions total forty thousand tokens and your agent runs a ten-step task, you have paid for those forty thousand tokens roughly ten times, once per model call, since the tool list is part of the request on every step. Multiply by every request across every user and the tool schemas alone can be a larger line on the bill than the actual conversation. None of that spend is doing work. The task needed three tools and you paid to describe two hundred.

The second cost is quieter and worse. Model tool selection degrades as the candidate set grows, and it degrades fastest when the tools are similar. Give a model search_tickets, search_customers, search_orders, search_docs, and search_inventory and the description text overlaps so heavily that the deciding detail is one clause buried in the middle of each. This is the same lost-in-the-middle problem that hurts long-context retrieval, pointed at your tool list. The model picks a plausible neighbor, calls the wrong search, gets empty or irrelevant results, and now you are paying for a recovery loop on top of the bloat. The wrong-tool call is not free either. It runs, it returns, and the model has to notice it was wrong and try again.

Both costs have the same root: you are shipping the whole toolbox when the task needs a handful of tools. So stop shipping the whole toolbox.

Retrieve tools the way you retrieve documents

The move is to treat tools as a retrieval problem. You already do this for knowledge. You do not paste your entire document store into the prompt; you embed the query, pull the most relevant chunks, and hand the model those. Do exactly the same thing with tools. Index every tool by its description, and on each turn, retrieve the small set whose descriptions match what the user is actually trying to do. The model sees ten relevant tools instead of two hundred, which cuts the token cost and makes the choice easy at the same time.

The index is small and cheap to build. Embed each tool's name and description once at startup and keep the vectors in memory. There is no reason to hit a vector database for a few hundred rows; a plain matrix and a dot product is faster and has no moving parts.

import numpy as np

class ToolIndex:
    """Embed every tool's name+description once at startup. A few hundred
    tools fit comfortably in memory, so retrieval is a single matmul, not a
    database round trip."""

    def __init__(self, tools: list[dict], embed):
        self.tools = tools
        self._embed = embed
        texts = [f"{t['name']}: {t['description']}" for t in tools]
        # L2-normalize so a dot product IS cosine similarity.
        matrix = np.array(embed(texts), dtype=np.float32)
        self.matrix = matrix / np.linalg.norm(matrix, axis=1, keepdims=True)

    def search(self, query: str, k: int = 8) -> list[dict]:
        q = np.array(self._embed([query])[0], dtype=np.float32)
        q /= np.linalg.norm(q)
        scores = self.matrix @ q            # cosine similarity to every tool
        top = np.argsort(scores)[::-1][:k]  # k best, highest first
        return [self.tools[i] for i in top]

Now the agent loop retrieves before it calls the model, and passes only the retrieved subset as the available tools.

async def run_turn(user_message: str, index: ToolIndex, history: list[dict]):
    # Retrieve on the live request, not the whole conversation, so the tools
    # track what the user is asking for RIGHT NOW.
    candidates = index.search(user_message, k=8)

    response = await model_call(
        messages=history + [{"role": "user", "content": user_message}],
        tools=[t["schema"] for t in candidates],  # 8 schemas, not 200
    )
    return response

That is the whole idea, and for a single-shot request it already works. You have taken the tool list from two hundred definitions to eight, the token cost drops by more than an order of magnitude, and the model is choosing from a short list of things that actually relate to the task.

The multi-turn problem: tools the task needs later

The naive version above retrieves on the current user message and throws the result away each turn. That breaks the moment a task spans several steps. A user asks the agent to "reconcile last month's invoices and flag the mismatches." The first turn retrieves invoice and accounting tools. Three steps in, the agent has found a mismatch and now needs to open a ticket, but create_ticket was never retrieved, because nothing in the original message mentioned tickets. The tool exists, the agent cannot see it, and the task stalls.

So retrieval on a live agent is not a per-turn lookup, it is a small working set you maintain across the run. You seed it with a few always-on core tools, add what each step retrieves, keep tools that were recently used because an active task tends to return to them, and evict the stale ones so the set stays small. It is a cache with a task-shaped eviction policy.

from collections import OrderedDict

class ToolWorkingSet:
    """A per-run working set of tools. Core tools are always present. Retrieved
    and recently-used tools fill the rest, capped so the model never sees more
    than `limit` at once."""

    def __init__(self, core: list[dict], limit: int = 12):
        self.core = {t["name"]: t for t in core}
        self.limit = limit
        self.active: OrderedDict[str, dict] = OrderedDict()

    def observe(self, retrieved: list[dict], just_used: list[str]):
        # Promote tools the model actually called: they move to the most-recent
        # end and survive eviction longest, because active tasks reuse tools.
        for name in just_used:
            if name in self.active:
                self.active.move_to_end(name)
        for tool in retrieved:
            if tool["name"] not in self.core:
                self.active[tool["name"]] = tool
                self.active.move_to_end(tool["name"])
        # Evict least-recently-used until we fit the budget (minus core).
        budget = self.limit - len(self.core)
        while len(self.active) > budget:
            self.active.popitem(last=False)

    def schemas(self) -> list[dict]:
        tools = list(self.core.values()) + list(self.active.values())
        return [t["schema"] for t in tools]

The core set is the detail that saves you. A handful of tools the agent needs constantly, a way to read state, a way to search, a way to escalate to a human, are marked core and never gated by retrieval. Everything in the long tail flows through the working set. This gives you a floor: even if retrieval whiffs completely on a turn, the agent still has its essential tools and can recover, instead of being stranded with a set that happens to miss the one thing it needs.

Give the model an escape hatch

Retrieval will sometimes miss, and the honest way to handle a miss is to let the model tell you. Add one meta-tool, always in the core set, that lets the agent search for a capability by intent when nothing it can see fits the job.

FIND_TOOL = {
    "name": "find_tool",
    "description": (
        "Search for a tool by describing what you need to do, when none of "
        "the available tools fit. Returns matching tools you can then call."
    ),
    "schema": {  # provider-specific tool schema wraps this
        "name": "find_tool",
        "parameters": {
            "type": "object",
            "properties": {"intent": {"type": "string"}},
            "required": ["intent"],
        },
    },
}

async def handle_find_tool(intent: str, index: ToolIndex, working: ToolWorkingSet):
    found = index.search(intent, k=5)
    working.observe(found, just_used=[])  # pull them into the visible set
    names = ", ".join(t["name"] for t in found)
    return f"Added tools: {names}. Call the one you need."

Now retrieval is a fast default, not a hard wall. Most turns the right tools are already visible. On the turn where they are not, the model says what it is trying to do, the matching tools get pulled into the working set, and the next step can call them. You have turned a silent failure into a recoverable one.

Where this breaks

Retrieval quality is now part of your agent's reliability, which means the ways retrieval goes wrong are the ways your agent goes wrong. Three of them bite in practice.

The first is bad tool descriptions. Retrieval matches on the description text, so a tool described as "handles account operations" will not surface for a query about closing a subscription, because the words do not meet in embedding space. The fix is boring and it works: write tool descriptions for retrieval, in the user's language, listing what the tool is for and when to reach for it, not internal jargon. This is the highest-leverage thing you can do and most teams skip it. If retrieval is missing tools, look at the descriptions before you touch the index.

The second is the recall floor. Pure semantic search misses when the query and the description share meaning but not vocabulary, the same reason dense retrieval alone underperforms on documents. If your miss rate is too high, the answer is the same one that works for document hybrid search with reranking: blend a keyword signal with the embedding score so an exact name match cannot be ranked out by a fuzzy semantic neighbor. Tools have short, distinctive names, and keyword matching on them is cheap insurance.

The third is eviction thrash. Set the working set too small and a multi-step task will evict a tool on one step and re-retrieve it two steps later, paying the retrieval cost repeatedly and occasionally missing on the re-fetch. Keep recently-used tools sticky, as the working set above does by promoting anything the model just called, and give the budget enough headroom that a normal task does not churn. This is the same instinct behind not aggressively pruning tool observations mid-task: dropping context that is about to be needed again is a false economy.

And measure it. Retrieval that you do not evaluate is a silent single point of failure. Build a labeled set of real tasks with the tools each one requires, and track recall at K, the fraction of the time the needed tools land in the retrieved set. That number, not the token savings, tells you whether K is high enough and whether your descriptions are pulling their weight. Tune against recall first, then trim K for cost once recall is where you want it.

The takeaway

The fat toolbox is a scaling failure disguised as an integration win. Every server you connect makes the agent more capable in principle and more expensive and less accurate in practice, because you are shipping the entire toolbox on every turn and asking the model to find one right answer in a crowd of look-alikes. Treat tools as something you retrieve, not something you always carry. Index them by description, retrieve the relevant few per turn, hold a small working set across a multi-step run, keep the essentials always on, and give the model a way to ask for what it cannot see. You get the token cost of a small toolbox and the reach of a large one, which is the only version of this that scales. It pairs naturally with the code execution pattern for MCP tools, which attacks the same bloat from the invocation side.

If your agent has quietly grown past fifty tools and you have never checked what that is doing to your bill or your selection accuracy, that is worth measuring before it becomes the reason your agent feels slower and dumber than it used to. Book a consultation call and we can look at what your agent actually reaches for versus what it carries.