Context Compaction for Long-Running AI Agents

Your agent works flawlessly in the demo. Then a customer runs it on something real, forty tool calls deep into a codebase migration, and it falls over. Maybe it blows past the context window. Maybe it starts hallucinating decisions it supposedly made an hour ago. Maybe it just quietly triples your token bill on every single turn. This is the number-one way production agents break, and the cause is almost always the same: context that grows without limit.

The lazy fix is to truncate. Lop off the oldest messages once you get near the ceiling. It's also the fastest way to make your agent forget the plan it committed to back on turn three. The better answer is compaction: you squeeze older context down into a dense running summary and keep the recent turns exactly as they were. This post covers why the business case is sharper than most people expect, a compaction loop you can drop into any agent today, and the newer server-side compaction that does the work for you.

The business problem hiding inside "it works on my machine"

Every turn of an agent loop resends the whole conversation. The system prompt, every tool definition, every prior tool call, and every one of those (often enormous) tool results. And you pay for input tokens on every request. So a conversation that reaches 150K tokens doesn't cost you 150K tokens once. It costs you 150K tokens on this turn, and again on the next, and again on the one after that. A 60-turn agent that grows linearly can burn through millions of input tokens finishing a single task.

Three costs pile up here:

• Money. Input tokens are where agent spend actually goes. At Opus pricing ($5 per million input tokens), an agent averaging 120K tokens of context across 50 turns spends around $30 on input before it writes a single output token.
• Latency. Time-to-first-token tracks prompt size. A bloated context makes every turn feel sluggish, and sluggish agents get closed.
• Reliability. This is the one teams tend to miss. A long, messy history causes context drift. The model loses track of what it already decided, repeats work it finished, or flatly contradicts itself. In practice, more agents fail from drift and lost memory on multi-step tasks than from running out of tokens.

Compaction hits all three at once. A smaller, better-organized context is cheaper to send, faster to process, and easier for the model to actually reason over.

Anchored summarization: the pattern that actually holds up

The approach that reliably beats both truncation and re-summarizing-everything is anchored iterative summarization. The idea is simple:

1. Keep one persistent anchor summary of everything old.
2. Keep the last N turns exactly as they happened, so recent detail is never lossy.
3. When the total crosses a token threshold, fold the turns about to age out into the anchor. Merge them in. Don't rebuild the summary from scratch.

That merge step is the whole trick. Folding new events into a running summary, instead of re-summarizing the entire history every time, keeps continuity intact and stops the summary itself from drifting the way the raw history would.

Here's a small but runnable version against the Anthropic API. It leans on the token-counting endpoint to decide when to compact, and never on a client-side guess like tiktoken, which mis-counts Claude tokens badly.

# pip install anthropic
from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-4-8"
COMPACT_THRESHOLD = 120_000   # tokens of history before we compact
KEEP_RECENT_TURNS = 6         # turns kept verbatim (never summarized)

def count_tokens(system: str, messages: list) -> int:
    # Model-specific and accurate. The only correct way to measure.
    return client.messages.count_tokens(
        model=MODEL, system=system, messages=messages
    ).input_tokens

def summarize(anchor: str, aging_out: list) -> str:
    """Fold the turns about to age out INTO the existing anchor summary."""
    transcript = "\n\n".join(
        f"{m['role'].upper()}: {_flatten(m['content'])}" for m in aging_out
    )
    resp = client.messages.create(
        model=MODEL,
        max_tokens=2000,
        system=(
            "You maintain a running summary of an agent session. "
            "Merge the new events into the existing summary. Preserve: "
            "decisions made, files/resources touched, open TODOs, and any "
            "facts the agent will need later. Be dense; drop chit-chat."
        ),
        messages=[{
            "role": "user",
            "content": (
                f"<existing_summary>\n{anchor or '(none yet)'}\n</existing_summary>\n\n"
                f"<new_events>\n{transcript}\n</new_events>\n\n"
                "Return the updated summary only."
            ),
        }],
    )
    return "".join(b.text for b in resp.content if b.type == "text")

def _flatten(content) -> str:
    if isinstance(content, str):
        return content
    parts = []
    for block in content:
        if block.get("type") == "text":
            parts.append(block["text"])
        elif block.get("type") == "tool_result":
            parts.append(f"[tool_result] {block.get('content')}")
        elif block.get("type") == "tool_use":
            parts.append(f"[tool_use {block.get('name')}] {block.get('input')}")
    return " ".join(str(p) for p in parts)

The compaction step runs before each model call. It checks the budget, and if we're over it, it summarizes the older turns and rewrites history down to the anchor summary plus the recent turns:

def maybe_compact(anchor: str, messages: list, base_system: str):
    """Returns (new_anchor, new_messages). Call before every model request."""
    system = base_system + (f"\n\n<session_summary>\n{anchor}\n</session_summary>" if anchor else "")
    if count_tokens(system, messages) < COMPACT_THRESHOLD:
        return anchor, messages

    # Split: everything except the last N turns is eligible to be folded in.
    recent = messages[-KEEP_RECENT_TURNS:]
    aging_out = messages[:-KEEP_RECENT_TURNS]
    if not aging_out:
        return anchor, messages   # nothing safe to compact yet

    new_anchor = summarize(anchor, aging_out)
    # History is now just the recent, verbatim turns; the rest lives in the anchor.
    return new_anchor, recent

Wiring it into an agent loop is boring, which is exactly the point. Compaction is a preprocessing step, not a rewrite of your control flow:

def run_turn(anchor, messages, base_system, tools, user_input):
    messages = messages + [{"role": "user", "content": user_input}]
    anchor, messages = maybe_compact(anchor, messages, base_system)

    system = base_system + (f"\n\n<session_summary>\n{anchor}\n</session_summary>" if anchor else "")
    resp = client.messages.create(
        model=MODEL, max_tokens=8000, system=system, tools=tools, messages=messages,
    )
    # Append the FULL content (tool_use blocks included), not just text.
    messages.append({"role": "assistant", "content": resp.content})
    return anchor, messages, resp

Two things decide whether this works. First, keep the recent turns lossless. The detail that gets summarized away is exactly what the model needs to finish the current step, so hold 4 to 8 turns verbatim as a safe default. Second, put the anchor where it caches well. Append it to the stable system prompt so your prompt cache prefix survives across turns. If you splice the summary in at the front of the prompt instead, you'll invalidate the cache every time it changes, which quietly undoes a chunk of the savings you were chasing.

The managed alternative: server-side compaction

Rolling your own buys you full control over what survives, which is invaluable when certain domain facts can never be dropped. But if you'd rather not babysit summarization logic, the Anthropic API now has server-side compaction in beta. You opt in with a beta header and a context-management directive, and the API condenses earlier context on its own as you near the trigger threshold:

messages = []

def chat(user_message: str) -> str:
    messages.append({"role": "user", "content": user_message})
    resp = client.beta.messages.create(
        betas=["compact-2026-01-12"],
        model="claude-opus-4-8",
        max_tokens=8000,
        messages=messages,
        context_management={"edits": [{"type": "compact_20260112"}]},
    )
    # CRITICAL: append the full content, including any compaction blocks.
    # The API uses those blocks to replace the compacted history next turn.
    messages.append({"role": "assistant", "content": resp.content})
    return "".join(b.text for b in resp.content if b.type == "text")

There's one rule that trips everyone up: append resp.content back to your message list whole. The response carries compaction blocks that the API uses to rebuild the condensed history on the next request. Pull out just the text and append that, and you silently throw away the compaction state, so the feature does nothing at all. It's the most common integration bug people hit with the managed API.

Reach for server-side compaction when you want the win without the upkeep. Stick with the DIY anchored version when you need to guarantee specific facts survive, want the summary observable and auditable, or you're on a stack where the managed feature isn't available yet.

Tradeoffs and pitfalls

Compaction is lossy by design. Plan for it:

• Summaries can drop the one fact you needed. Spell out in the summary system prompt what must always survive (decisions, IDs, file paths, open TODOs), and keep enough recent turns verbatim to cover the gap.
• Compaction costs tokens of its own. Each summarization is an extra model call. On long sessions it pays for itself many times over. On short ones it's pure overhead, so only compact once you actually cross the threshold, never on a fixed timer.
• Never compact mid-tool-call. If the model just emitted a tool_use block and you haven't sent back the tool_result, rewriting history can orphan the call. Compact at turn boundaries, once results are in.
• Make it observable. Log what went into each request, what got compacted, and the summary at every step. When an agent wanders off three hours into a run, you'll want to see the exact context it was reasoning over instead of guessing at it.

The takeaway

Long-running agents don't fall over because the model is weak. They fall over because context grows faster than anyone planned for, and the obvious fix, truncation, happens to be the one that wrecks reliability. Anchored summarization keeps recent detail intact while folding old history into a dense, cache-friendly anchor, which cuts cost and latency without inviting drift. Server-side compaction gets you most of the way there with a header and one careful append. Whichever you pick, the mindset is the same: treat context as something you actively manage, not a buffer you let fill up on its own.

If you're building an agent that needs to run for hours without falling over, or you're already debugging one that doesn't, book a consultation call and we'll pressure-test your context strategy together.