Your Tool Returned 40,000 Tokens. The Agent Needed 12.

A tool call comes back with a JSON response from an internal API. It is 38,000 tokens. The agent looked at it to answer one question: is the account past due. That answer is a single boolean buried in field 14 of a nested object. The other 37,990 tokens were addresses, historical line items, tax metadata, none of which the task needed.

That would be merely wasteful if it happened once. It does not happen once. The blob is now in the message history, and it stays there. Every subsequent turn of the agent re-sends those 38,000 tokens to the model, because the model has no memory between calls other than the context you hand it each time. A ten-step agent that pulled three big responses like this early on is paying for all three, on all ten steps, whether or not it ever looks at them again. The window fills with sediment, prefill gets slower, and eventually you either hit the context limit or start truncating the very history the agent needs to stay coherent.

The fix is not to summarize the blob after the fact. It is to never put the blob in context in the first place.

The pattern: hand back a handle, not the payload

The idea is old and comes from the filesystem. When a program opens a large file, it does not load the whole thing into a variable and pass it around by value. It gets a file descriptor, a small handle, and reads the parts it needs when it needs them. Agent tool results should work the same way.

When a tool produces a large output, write the full payload to a store, the local filesystem, an object store, a key-value cache, whatever fits your deployment. Return to the model a small, structured reference: an id it can use to fetch the payload later, plus a preview that tells it what is inside. The model sees the shape of the data and decides whether it needs to read more. Most of the time the preview is enough and it moves on. When it genuinely needs a specific slice, it calls a read tool with the handle and asks for exactly that slice.

This is context offloading, and in 2026 it is how the serious long-running agents (coding agents, deep research agents) keep going for hundreds of steps without drowning in their own tool output. The payload lives in durable storage. The active context holds pointers.

Intercept large results at the tool boundary

You do not want to rewrite every tool to think about this. Wrap them. The wrapper runs the tool, measures the result, and decides whether it is small enough to pass through inline or large enough to offload.

import json
import uuid
from pathlib import Path

STORE = Path("/var/agent/artifacts")   # any durable, readable location
INLINE_LIMIT = 2000                     # chars; roughly ~500 tokens

def estimate_tokens(text: str) -> int:
    # Cheap heuristic; swap in your real tokenizer if you want precision.
    return len(text) // 4

def offload_if_large(tool_name: str, raw_output: str) -> dict:
    # Small results pass straight through. No indirection, no round-trip.
    if len(raw_output) <= INLINE_LIMIT:
        return {"inline": True, "content": raw_output}

    # Large results go to the store and come back as a handle + preview.
    artifact_id = f"{tool_name}-{uuid.uuid4().hex[:8]}"
    STORE.mkdir(parents=True, exist_ok=True)
    (STORE / f"{artifact_id}.txt").write_text(raw_output)

    return {
        "inline": False,
        "artifact_id": artifact_id,
        "approx_tokens": estimate_tokens(raw_output),
        "preview": build_preview(raw_output),
    }

The whole decision is one length check. Under the limit, the model sees the content as before and nothing changes. Over the limit, the model sees a handle and a preview instead of the wall of text.

The preview is the part that matters

A handle with no preview is useless, because the model cannot decide whether to fetch something it cannot see. The preview has to carry enough structure for the model to answer two questions: do I need the full payload, and if so, which part. For structured data, give it the shape. For text and logs, give it the ends and the size.

def build_preview(raw_output: str) -> dict:
    # Try structured first: expose the shape, not the values.
    try:
        data = json.loads(raw_output)
    except json.JSONDecodeError:
        # Text or logs: first and last lines plus a total count is
        # usually enough for the model to target a slice.
        lines = raw_output.splitlines()
        return {
            "kind": "text",
            "total_lines": len(lines),
            "head": lines[:8],
            "tail": lines[-4:],
        }

    if isinstance(data, dict):
        return {"kind": "object", "keys": list(data.keys())}
    if isinstance(data, list):
        return {
            "kind": "array",
            "length": len(data),
            "sample": data[:2],   # a couple of rows to show the row shape
        }
    return {"kind": "scalar", "value": data}

Now the model, instead of 38,000 tokens of account JSON, sees something like {"kind": "object", "keys": ["id", "status", "past_due", "line_items", ...], "approx_tokens": 9200, "artifact_id": "get_account-3f9a1c22"}. If the task is "is this account past due," the model may not even need to fetch. past_due is right there in the key list, and it can call a read tool for that one field. The 38,000 tokens never touch the window.

Give the model a way to read back

Offloading only works if the model can retrieve the parts it needs. That means a companion tool, exposed alongside your normal tools, that takes a handle and a way to address a slice. Keep the addressing simple and composable: a line range for text, a key path for structured data.

def read_artifact(artifact_id: str, path: str | None = None,
                  start: int = 0, count: int = 40) -> str:
    """Fetch a slice of an offloaded artifact.

    path:  dotted key path into JSON, e.g. "line_items.0.amount".
    start/count: line window for text, applied when path is absent.
    """
    blob = (STORE / f"{artifact_id}.txt").read_text()

    if path:
        node = json.loads(blob)
        for key in path.split("."):
            node = node[int(key)] if key.isdigit() else node[key]
        return json.dumps(node)

    lines = blob.splitlines()
    window = lines[start:start + count]
    return "\n".join(window)

Register read_artifact in the same tool schema the model already sees, with a description that tells it how the handle and slice work. The model learns the loop quickly: it gets a preview, decides it needs line_items, calls read_artifact(artifact_id, path="line_items"), and pulls only that array. You have turned a 38,000 token dump into a 9,200 token preview and a targeted read of maybe 3,000 tokens, and only the parts it actually pulled ever stay in context.

The pitfalls that actually bite

A thin preview is worse than no offloading. If the preview does not tell the model what is inside, it will either fetch the whole payload back (you gained nothing and added a round-trip) or guess wrong and answer from incomplete data. Spend your effort here. The preview is a table of contents for the payload, and its job is to make the fetch decision obvious. Test it by asking whether you, seeing only the preview, would know what to fetch.

Do not offload things the model needs every single turn. Offloading trades a resident cost for a fetch cost. That is a win when the payload is large and consulted rarely. It is a loss when the payload is small or the model needs it on every step, because then you are paying the round-trip repeatedly to save tokens you were going to use anyway. The length threshold handles the small case. Watch for the second case in your traces: if the model fetches the same artifact on every turn, it should have been inline.

Handles go stale, and a stale handle is a correctness bug. If a later tool call changes the underlying resource, the artifact you offloaded is now an old snapshot, and a model that fetches it is reading the past as if it were the present. This is the same failure I wrote about in stale tool observations: the agent is not confused, its context is out of date. Key artifacts by the resource they describe and let a newer write supersede the older handle, so a read always resolves to current state.

Offloading and compaction are different tools, use both. Offloading keeps the big payloads out of the window at the tool boundary. Context compaction shrinks the running conversation once it grows long. They stack: offload the blobs so they never bloat the history, then compact the history so the reasoning trail stays affordable over a long run. Doing only one leaves the other source of bloat untouched.

Storage needs a lifecycle. Artifacts accumulate. For a short-lived request that is fine, the process ends and the temp directory goes with it. For a long-running or resumable agent, put a TTL or a per-session namespace on the store and clean it up, or you will slowly fill a disk with the JSON of every API call every agent ever made.

The takeaway

The default agent loop treats every tool result as something to pour directly into the model's context, in full, forever. For small results that is fine. For the large ones, a file, a verbose API response, a page of logs, it means you pay for tens of thousands of tokens on every turn after the one that needed them, and you march the context window toward its limit for no benefit. Offloading flips it: the payload goes to a store, a compact handle and a real preview go to the model, and the model fetches the specific slice it needs, when it needs it. It is a wrapper at the tool boundary and one extra read tool, and on a tool-heavy agent it is one of the largest cost and context wins you can make without touching the model or the prompt.

If your agents are burning tokens on tool output they barely read, or hitting the context ceiling on long runs, this is exactly the kind of production work I help teams sort out. Book a consultation call and we can find where your context is filling up with sediment.