Your Agent's Memory Is Full and Most of It Is Junk

Give an agent persistent memory and the first month is a demo that sells itself. It remembers the user's name, their timezone, the fact that they are on the enterprise plan, and it feels like the product finally knows them. Then six months pass. The same user has had two hundred sessions, every one of them wrote a few facts, and nothing ever removed anything. Now the store holds the user's timezone recorded eleven times, three contradictory answers to "what plan are they on" because they upgraded twice, and a pile of vague paraphrases like "user seemed frustrated with billing" that were never precise enough to act on and never will be. Retrieval runs over all of it, and quietly, week by week, the agent gets worse at the exact thing memory was supposed to make it good at.

This is not a storage problem. Disk is cheap and a vector index will hold millions of rows without complaint. It is a garbage-collection problem, and it is getting enough attention now that OWASP added memory and context poisoning to its 2026 top ten for agentic applications. Audits of popular memory setups keep finding the same thing: the majority of what is stored is effectively dead weight, and a large share of every retrieval is tokens the model never uses. If you built memory the way most tutorials teach it, as an append-only extract-and-store loop, you have this problem right now whether or not anyone has noticed.

The business problem: accumulation is not memory

There is a real distinction hiding behind a word everyone uses loosely. Memory is a curated set of true, useful facts. Accumulation is every fact the system ever wrote, in every version, including the dead ones. Retrieval over an accumulation returns everything the agent ever believed, and the model is left to guess which version is current. Sometimes it guesses wrong, and because the wrong answer came from "memory" it states it with total confidence.

The cost shows up in three places. The first is dollars: every duplicate and every stale row is a candidate for retrieval, so you pull them into context, embed them, and pay to drag tokens the model ignores through every turn. The second is quality, which is worse because it is invisible. A stale fact often outranks the fresh one simply because it was recorded more times, so the more a user interacts, the more the old belief gets reinforced. The third is trust. The single fastest way to make a user stop believing your agent knows them is to have it confidently repeat something that was true four months ago and is now wrong. You built memory to earn trust, and an uncurated store spends it.

If you have not built the persistence layer yet, start there first, because curation is a discipline you add on top of a working extract, store, retrieve loop for persistent agent memory. This post assumes you have that and are now watching its quality decay. The fix has four parts: gate what gets in, reconcile and dedupe on the way in, decay and evict what stops earning its place, and measure whether any of it is working.

Step one: admission control at write time

The cheapest junk to deal with is the junk you never store. Most extraction pipelines are too eager. They run an LLM over the transcript, ask "what should I remember," and dutifully save whatever comes back, including transient details and low-signal observations that will never be worth a retrieval slot. The first gate is an admission policy that decides whether a candidate fact is durable and specific enough to keep at all.

from dataclasses import dataclass, field
from datetime import datetime, timezone
import time

@dataclass
class MemoryCandidate:
    subject: str          # the entity this fact is about, e.g. "user:42"
    predicate: str        # a normalized key, e.g. "plan", "timezone"
    value: str            # the fact itself
    confidence: float     # extractor's confidence, 0..1
    source_session: str

# Facts worth keeping are durable (still true next month), specific
# (a concrete value, not a mood), and confidently extracted. Everything
# else is noise that will only dilute retrieval later.
TRANSIENT_PREDICATES = {"current_mood", "last_message", "session_intent"}

def admit(candidate: MemoryCandidate) -> bool:
    if candidate.predicate in TRANSIENT_PREDICATES:
        return False                     # ephemeral by nature, never persist
    if candidate.confidence < 0.6:
        return False                     # the extractor was guessing
    if len(candidate.value.strip()) < 3:
        return False                     # too vague to ever match a query
    return True

The point is not the exact thresholds, which you will tune. The point is that admission is a decision you make on purpose, with a default of no, rather than storing everything and hoping retrieval sorts it out. A fact about a user's mood in one session is real, but it is not memory, it is a log line, and logs belong somewhere cheaper than your retrieval path.

Step two: reconcile and deduplicate on the way in

Admission stops obvious noise, but it does not stop the same true fact from being written eleven times, or a new value from silently contradicting an old one. Before you insert, look for what you already hold about the same subject and predicate, and decide whether this is a duplicate to drop, an update to supersede the old value, or a genuinely new fact. This is where identity resolution matters: if the same user is stored under user:42 in one session and customer_42 in another, you have split their history and neither half is complete, so normalize the subject key before you compare.

def upsert(store, candidate: MemoryCandidate) -> str:
    subject = normalize_subject(candidate.subject)   # collapse aliases first
    existing = store.get_facts(subject=subject, predicate=candidate.predicate)

    for row in existing:
        if row.value == candidate.value:
            store.touch(row.id)          # exact duplicate: bump recency, do not add a row
            return "duplicate"
        if near_duplicate(row.value, candidate.value):
            store.touch(row.id)          # semantic dup ("PST" vs "Pacific Time")
            return "near_duplicate"

    if existing:
        # A different value for the same key is a contradiction, not an addition.
        # Newer, confidently extracted facts supersede older ones for single-valued
        # predicates like "plan"; keep the old row marked superseded for audit.
        store.supersede(existing, by=candidate)
        return "superseded"

    store.insert(candidate)
    return "inserted"

Two things make this work. First, single-valued predicates like plan or timezone can only have one live value, so a new one supersedes rather than adds, which is what kills the "stale outranks fresh" failure. Multi-valued predicates like owns_product are allowed to accumulate, but still deduped. Second, you mark superseded rows rather than hard-deleting them immediately, so retrieval never sees them but an audit can, which means a bad supersede is recoverable. This is the same instinct as consolidating a new fact against what you already hold, extended from one fact to the whole subject and made explicit about contradictions.

Step three: decay and evict what stops earning its slot

Some entries are not wrong and not duplicated, they are just rarely useful, and they still cost you a retrieval candidate every single query. Hard-deleting them is too blunt, because "rarely useful" is not "never useful." The better tool is a decay score that combines recency, how often the entry was actually retrieved and used, and its confidence, so an entry that never earns a hit slowly loses weight and eventually falls below an eviction floor.

import math

def memory_score(row, now: float) -> float:
    age_days = (now - row.created_at) / 86400
    idle_days = (now - row.last_used_at) / 86400

    # A forgetting curve, not a cliff: value decays with idle time but is
    # rescued every time the memory actually gets used in an answer.
    recency = math.exp(-idle_days / 30.0)          # ~half-life of 3 weeks idle
    usefulness = math.log1p(row.use_count) / 3.0   # rewards repeated real use
    return row.confidence * (0.6 * recency + 0.4 * usefulness)

def sweep(store, now: float, floor: float = 0.15) -> int:
    evicted = 0
    for row in store.iter_live_facts():
        if row.pinned:                 # some facts are load-bearing; never evict
            continue
        if memory_score(row, now) < floor:
            store.evict(row.id, reason="decayed_below_floor")   # audit row kept
            evicted += 1
    return evicted

Run this as a periodic sweep, not on the hot path, so it never adds latency to a user turn. The two details that matter in practice: rescue an entry's score whenever it is actually used, so genuinely useful facts survive indefinitely no matter how old they are, and let callers pin facts that must never decay, like a hard compliance flag or an explicit user preference. Decay is how you get the effect of forgetting without the risk of amnesia, and it keeps the retrieval candidate set small, which is the same reason pruning stale observations keeps an agent's working context clean.

Step four: measure it, or you are flying blind

Every step above has a knob, and you cannot tune knobs you cannot see. Instrument the store so you know whether curation is keeping up, because "memory feels worse lately" is not something you can act on. Three numbers tell you almost everything.

def health_report(store) -> dict:
    live = list(store.iter_live_facts())
    by_subject = group_by(live, key=lambda r: (r.subject, r.predicate))

    dup_groups = sum(1 for rows in by_subject.values() if len(rows) > 1)
    contradictions = sum(
        1 for (subj, pred), rows in by_subject.items()
        if is_single_valued(pred) and len({r.value for r in rows}) > 1
    )
    # Of memories retrieved into context, how many the model actually used.
    hit_rate = store.used_retrievals() / max(1, store.total_retrievals())

    return {
        "live_facts": len(live),
        "duplicate_groups": dup_groups,      # should trend toward zero
        "contradictions": contradictions,    # single-valued keys with >1 live value
        "retrieval_hit_rate": round(hit_rate, 3),  # used / retrieved
    }

Watch the retrieval hit rate most closely. If you are pulling memories into context and the model uses a small fraction of them, you are paying to poison your own prompt with distraction, and it is a leading indicator of both rising cost and falling answer quality. A rising duplicate or contradiction count means your write-time reconciliation is not aggressive enough. These numbers turn "memory quality" from a vibe into a dashboard you can defend.

The tradeoffs, and where it bites

Curation is not free, and pretending otherwise is how you trade one problem for another. Every write now does a read and a comparison, and semantic dedup may cost an embedding or a small model call, so batch the reconciliation at end of session rather than after every message if your agent is chatty. The decay sweep is background work, but on a large store it is real compute, so shard it and run it off peak.

The sharper risk is over-eviction. A decay floor set too high, or a supersede rule that is too trigger-happy, will throw away a fact the user very much expected the agent to keep, and that failure is more damaging than a bit of clutter because it reads as the agent forgetting something it was explicitly told. This is exactly why every removal writes an audit row instead of vanishing: you want a bad eviction to be diagnosable and reversible, and you want to tune the floor from real retrieval data rather than a guess. Start conservative, watch the health report, and tighten only when the numbers say the store is bloating faster than it is being used.

One more trap. Superseding on contradiction assumes the newer fact is the true one, which is usually right but not always, because extraction makes mistakes and a low-confidence new value should not overwrite a high-confidence old one. Weight the supersede decision by confidence, keep the loser marked rather than deleted, and you get the freshness benefit without letting a single bad extraction rewrite the user's history.

The takeaway

Persistent memory is not something you build once and leave running. An append-only store does not get smarter as it fills, it gets noisier, and the decay in retrieval quality is invisible right up until a user catches the agent confidently repeating something that stopped being true months ago. Treat the store the way you would treat any long-lived dataset that other systems read from: gate what gets written so noise never enters, reconcile and dedupe on the way in so one fact is stored once and contradictions resolve instead of pile up, decay and evict what stops earning its slot so retrieval stays small and sharp, and measure duplicate rate, contradiction rate, and retrieval hit rate so you can see the store's health instead of guessing at it. Do that and memory keeps the property you built it for, which is that the agent genuinely knows the user, rather than slowly drowning in everything it ever heard.

If your agent has had persistent memory running for a while and you suspect its recall is quietly getting worse, that is usually measurable in an afternoon and fixable soon after. Book a consultation call and we can look at what your memory store actually holds and whether a curation layer like this is the cleanest way to get its quality back.