Your Agent Compacts on a Token Count, Not on What It Needs
Context compaction is sold as a cost fix, but once you are compacting, two decisions quietly run your agent: when the compaction fires and what it keeps. Inherit the defaults, a token threshold and keep-the-last-N turns, and your agent evicts the plan it committed to while preserving its own chit-chat.
A team I was helping had an agent doing a framework migration across a mid-size repo. It ran fine for the first hour. Then, about eighty tool calls in, it rewrote a config file it had already fixed on turn four, undoing the exact decision it had committed to at the start. Nothing crashed. The token bill looked healthy. When we pulled the trace, the cause was almost funny: the harness had compacted the history when it crossed a 120K token threshold, and the summary it kept had dropped the migration plan while faithfully preserving the last six turns of the model asking itself what to do next. The agent had not run out of room. It had thrown away the one thing it needed and kept the noise.
Compaction gets framed as a cost problem, and at first it is. But once you are compacting, two quieter decisions start running your agent, and almost nobody chooses them on purpose: when the compaction fires, and what it keeps. Both usually arrive as defaults you inherited from a tutorial, a fixed token threshold and keep-the-last-N turns, and both are wrong in ways that only surface on the long sessions you actually care about. This essay is about the compaction policy, not the compaction mechanics, and what the current research says about choosing it on purpose.
Compaction hides two decisions nobody made
If you have already added anchored compaction to an agent, you solved the mechanism: old turns fold into a running summary, recent turns stay verbatim, context stops growing. That is the easy half. The hard half is the policy wrapped around it, and it is made of two decisions that most codebases express as magic numbers.
The first is the trigger. COMPACT_THRESHOLD = 120_000 says compact when the history crosses 120K tokens. That is a decision about when, expressed as a number that has nothing to do with what the agent is doing at that moment.
The second is the eviction rule. KEEP_RECENT_TURNS = 6 says keep the last six turns and summarize everything before them. That is a decision about what, expressed as a bet that recency is a good proxy for importance.
Neither number is obviously wrong, which is exactly why they survive code review and then quietly misbehave in production. The trigger fires on the clock. The eviction rule keeps whatever happened most recently. On a short task you never notice, because the task finishes before either decision gets tested. On a long one, both get tested constantly, and the failures look like a weak model rather than a policy you never chose.
A token threshold fires on the clock, not the task
A fixed token threshold has one job, bound the context size, and it does that job without ever asking whether this is a safe moment to lose detail. Two failure shapes follow from that blindness.
It fires too early. The threshold trips in the middle of a multi-step operation, right when the model is holding the most working detail it will need. You compact, the detail goes into a summary, and the model finishes the step from a lossy paraphrase of what it was doing a second ago. This is the most expensive kind of drop because the evicted detail was load-bearing for the very next action.
It fires too late. The threshold is set high to avoid over-compacting, so by the time it trips the context is already large enough that recall has degraded. Anthropic frames the context window as a finite resource with diminishing marginal returns, and the needle-in-a-haystack work it cites shows recall weakening well before the window is full. A threshold that only fires near the ceiling has already let context rot set in before it does anything about it.
The fix is not a better number. It is to make the trigger aware of task state. Compact at task boundaries, after a result comes back and before the next plan step, never mid-tool-call, and let a soft token budget decide whether rather than a hard threshold decide now. The budget bounds cost. The boundary decides timing. They are two different jobs and a single magic number was doing both badly.
Recency is the wrong thing to protect
Keeping the last N turns verbatim assumes the newest content is the most important. On an agent loop that assumption inverts the longer the session runs, because the most important content, the plan, the decisions, the IDs, tends to arrive early and then sit in the weakly attended middle of the context while the recent turns fill up with the model narrating its own reasoning.
So a recency rule keeps the narration and evicts the plan. The lost-in-the-middle effect then compounds it: even before compaction, a decision stranded in the middle is already being under-weighted, and compaction finishes the job by summarizing it away. This is the same failure that makes keying observations by the resource they describe beat an append-only log, applied to compaction instead of pruning.
The alternative is task-aware retention: protect a pinned set explicitly, and evict the rest by usefulness rather than age. In enterprise tool-use workflows, a Microsoft team found that pruning older tool interactions plus compact summarization reached 91.6% complete task success against 71.0% for keeping the full history, while using fewer tokens. Selective retention beat retention. The policy below keeps a pinned set, protects a small recent window, and evicts the least recently referenced, biggest payloads first.
# A compaction policy that protects a pinned set and evicts by score, not age.
from dataclasses import dataclass
@dataclass
class Turn:
id: str
tokens: int
pinned: bool # decisions, IDs, the plan: never evict
last_used_turn: int # when the model last referenced this item
is_event: bool # a real event vs a superseded observation
def select_for_compaction(turns, budget, keep_recent=4):
recent, pool = turns[-keep_recent:], turns[:-keep_recent]
# Never touch the pinned set or the protected recent window.
evictable = [t for t in pool if not t.pinned]
# Least useful first: oldest reference, then biggest payload.
evictable.sort(key=lambda t: (t.last_used_turn, -t.tokens))
kept = [t for t in pool if t.pinned] + recent
need = sum(t.tokens for t in turns) - budget
freed, to_evict = 0, []
for t in evictable:
if freed >= need:
kept.append(t)
else:
to_evict.append(t); freed += t.tokens
return kept, to_evict
The pinned flag is the important part, and it does not set itself. Mark a turn pinned when it records a decision, an identifier, a file path, or an open TODO, the things a summary is most likely to paraphrase into uselessness. Everything else competes for space on how recently the model actually touched it.
Let the model manage its own context
The harness guessing what to keep has a ceiling: the harness does not know what the model is about to do next. The model does. The newest line of work hands the controls to the model and lets it manage its own window.
In the CTXAgent study, researchers gave frontier models two tools and one paragraph of guidance, then let the model decide when to compact, what to keep, and what to write as the handoff summary. Across mixed sessions of 35 to 212 turns, the stronger models kept live context under 100K tokens without a significant drop in task success. With Claude Opus 5.5, the self-managed harness matched a fixed Claude Code configuration on success while costing 34% less than its explicit 200K-window setup, and needed no context-window setting at all.
The shape is simple: expose compaction as tools and describe when to use them.
# Give the model the controls and let it decide when and what to compact.
CONTEXT_TOOLS = [
{
"name": "compact_context",
"description": (
"Fold older turns into a summary when the working context gets "
"noisy. Keep decisions, IDs, file paths and open TODOs verbatim; "
"drop superseded tool output. Call this at a task boundary, not "
"in the middle of a step."
),
"input_schema": {
"type": "object",
"properties": {
"keep_ids": {"type": "array", "items": {"type": "string"}},
"summary": {"type": "string"},
},
"required": ["keep_ids", "summary"],
},
},
{
"name": "recall",
"description": "Fetch an evicted item back by id, no tool re-run.",
"input_schema": {"type": "object",
"properties": {"id": {"type": "string"}}, "required": ["id"]},
},
]
# The harness still enforces a hard ceiling: if live tokens exceed the cap,
# compact on the model's behalf rather than trust it to notice in time.
The caveat in that last comment is not optional. The same study found some models compacted so often that other costs rose, so self-management is a better default, not a free one. You keep a hard ceiling the harness enforces, and a pinned set the model is told it may not drop.
Keep an addressable way back
Every policy above is lossy, and lossy is fine as long as loss is reversible. The move that makes it reversible is to stop deleting evicted content and start addressing it. Instead of a similarity search over a vector store, an append-only log assigns every evicted item an id, and the agent asks for the full content back by that id when it needs it.
The ARC work builds exactly this: tool outputs go into an append-only log addressable by id, compaction replaces old outputs with compact references, and the agent retrieves full content by id without re-running the tool. On needle-in-a-haystack recall it reports 99.40% exact-answer accuracy against 88.12% for the best baseline, at lower serving cost. The point is not the benchmark, it is the architecture: eviction becomes a pointer swap, not a delete.
# Evicted content is not deleted. It moves to an append-only log, keyed by id.
class RecallStore:
def __init__(self):
self._log = {} # id -> full payload
def evict(self, turn):
self._log[turn.id] = turn.payload
# What stays in context is a stub the model can still act on.
return {"id": turn.id, "ref": f"[evicted {turn.id}: {turn.summary}]"}
def recall(self, id):
return self._log.get(id) # full content back, no tool re-run
store = RecallStore()
stub = store.evict(old_turn) # context shrinks, nothing is lost
full = store.recall(old_turn.id) # model pulls it back on demand
This pairs naturally with the habit of offloading big tool results to a store before they ever enter the window. The cheapest token to compact is one that was never in the context, and the safest token to evict is one you can still address.
The parts that bite
A policy you chose on purpose still has sharp edges. These are the ones that draw blood.
- Self-management is not self-regulating. A model deciding to compact is not a model that budgets compaction well. Keep the harness ceiling as a backstop, and log how often the model compacts so you can catch a model that is burning calls on it.
- A pinned set has to be enforced, not requested. Telling the model not to evict the plan is a suggestion. Giving the harness a list of ids it refuses to summarize is a guarantee. Use the guarantee for anything a task cannot recover from losing.
- Verify the survival, do not assume it. After each compaction, check that the pinned ids are still present in the assembled context. A one-line assertion turns a silent drop into a loud failure you can fix.
- Summaries are not memory. Anything that must outlive the session belongs in persistent memory the agent reads back, not in a summary the next compaction rewrites. If you are already curating and garbage-collecting that memory, the compaction policy is its short-term counterpart, not a replacement.
- Measure the policy, not just the mechanism. A compaction loop that passes its unit tests can still lose tasks. The number that matters is end-to-end task success on long sessions, before and after a policy change, not tokens saved.
The takeaway
Long agents do not fall over because the model is weak. They fall over because context grows faster than anyone planned, and the obvious defaults for managing it, a fixed token threshold and keep-the-last-N turns, happen to be the two that fail worst on exactly the sessions you care about. The threshold fires on the clock instead of the task. Recency keeps the narration and evicts the plan. Both were decisions, not constants, and nobody made them.
So make them. Trigger on task boundaries under a soft budget, protect a pinned set the harness enforces, keep an addressable log so eviction is reversible, and let the model help decide what to keep as long as a hard ceiling has the final say. If your agent works fine on short tasks but makes baffling decisions on long ones, the compaction policy is one of the first places worth looking, and it is usually fixable without touching the model at all. Book a consultation call and we can find where your agent is throwing away the thing it needed.
Questions this essay answers
01What is a context compaction policy?
It is the pair of decisions that run every time an agent compacts: the trigger, meaning when compaction fires, and the eviction rule, meaning what gets kept and what gets squeezed into a summary. Most teams never choose either one on purpose. They inherit a fixed token threshold and a keep-the-last-N-turns rule from a tutorial, and those defaults decide what the agent forgets. The mechanics of summarizing are easy. The policy is the part that determines whether the agent keeps the plan it committed to or throws it away to make room for noise.
02How is this different from compaction itself?
Compaction is the mechanism: fold old turns into a dense summary and keep recent detail so context stops growing. The policy sits on top of the mechanism and controls two levers the mechanism does not. The first is timing, because a token threshold fires on token count rather than on whether the task is at a safe boundary. The second is selection, because keeping the newest turns is not the same as keeping the useful ones. You can have a correct summarization loop and still lose tasks because the policy around it evicts the wrong things at the wrong moment.
03Can I trust the model to manage its own context?
Partly, and that is the honest answer. Recent work shows strong frontier models can decide when to compact and what to keep, and do it cheaper than a fixed window. But behavior varies by model, and some compact so aggressively that other costs climb. Treat self-management as a better default, not a free one. Keep a hard token ceiling the harness enforces no matter what the model does, keep a pinned set the model is not allowed to evict, and verify after each compaction that the non-negotiable facts actually survived.
04When should a dropped fact live in memory instead of a summary?
When it has to outlive the session. A summary is rewritten on every compaction, so anything you fold into it can drift or vanish on the next pass. Facts that must persist, a customer ID, a schema decision, an approval that was granted, belong in persistent memory the agent reads back on demand, not in a running summary that the next compaction can quietly overwrite. The rule of thumb: summaries hold the shape of what happened this session, memory holds the facts that must be true across all of them.
