You Upgraded Your Embedding Model and Silently Broke Retrieval

A new embedding model shipped this quarter. It scores higher on the retrieval benchmarks, supports a longer input, and costs less per million tokens. Someone on your team changes the model name in the config, redeploys, and the pipeline comes back green. No errors, no failed tests, no alerts. For a week everything looks fine, and then support starts forwarding tickets where the assistant confidently answers a question with a document that has nothing to do with it.

The change that broke retrieval is the one that looked safest. Swapping an embedding model is not a config edit. It is a data migration, and it is the kind that fails silently.

The business problem: two models, two coordinate systems, one index

An embedding model maps text into a vector space that only that model understands. The 1,536 numbers your old model produced for a document and the 1,536 numbers a new model produces for the same document are not two versions of the same address. They are addresses in two different cities. Cosine similarity between a query vector from the new model and a document vector from the old model is a real number the database will happily compute and rank, and it is meaningless.

So the moment you point queries at the new model while your index still holds vectors written by the old one, every search is comparing across incompatible spaces. Nothing errors. The database returns its top k as always. The results are just subtly wrong at first, then badly wrong as more of the drift compounds, and the only external symptom is answers grounded in the wrong sources. This is the failure people have started calling semantic rot, and it is worse than a crash because a crash pages someone.

There are two costs stacked here. The first is quality: your retrieval degrades exactly where you cannot see it, inside a similarity score that has no natural error signal. The second is money. Re-embedding is a real budget line now, not an afterthought. Fifty million chunks at a few hundred tokens each is on the order of tens of billions of tokens to re-embed, which is real compute hours and real dollars, and you pay roughly double your storage for the window where both indexes exist. A migration you do blindly can cost as much as it corrupts.

The fix is to treat the upgrade like what it is: a schema migration with a validation gate and a rollback, run against live traffic without a maintenance window. This is the same discipline I argued for when a new chat model drops and everyone wants to bump the string. Embeddings need it more, because the failure hides better.

Rule one: a vector without a model version is a liability

Before you migrate anything, make it impossible to mix spaces by accident. Every vector you store gets tagged with the exact model and dimension that produced it, and every query is pinned to one version. If a query for version B ever reaches vectors from version A, that is a bug you want to fail loudly, not a silent cross-space comparison.

from dataclasses import dataclass

# The embedding model identity is part of the data, not ambient config.
# Bump this deliberately; never let it float to "whatever is deployed".
@dataclass(frozen=True)
class EmbeddingVersion:
    model: str          # e.g. "text-embed-3-large"
    dim: int            # vectors of a different width can't even share an index
    metric: str         # "cosine" | "dot" | "euclidean", must match at query time

    @property
    def namespace(self) -> str:
        # One physical namespace per version. Vectors from two versions never
        # share a namespace, so a query can only ever compare like with like.
        return f"{self.model}__{self.dim}"


def embed_query(text: str, version: EmbeddingVersion) -> list[float]:
    vec = embedding_client.embed(text, model=version.model)
    if len(vec) != version.dim:
        # Guards against a provider silently changing default dimensions.
        raise ValueError(f"{version.model} returned {len(vec)}, expected {version.dim}")
    return vec

The namespace-per-version rule is the whole game. If your vector store supports named namespaces or collections (Pinecone, Qdrant, Milvus, pgvector with a version column all do), writing the new model into its own namespace means the old index keeps serving correct results the entire time you build the new one. There is no window where the two are blended.

Rule two: backfill into the new namespace, idempotently and resumably

Re-embedding your whole corpus is a long job that will be interrupted, so write it to be restarted without redoing finished work and without double-writing. Key each vector by a stable content id plus the version, batch the embedding calls, and record progress so a crash at chunk nine million resumes at chunk nine million.

def reindex(source_chunks, new_version: EmbeddingVersion, checkpoint, batch_size=256):
    """Re-embed the corpus into new_version's namespace. Safe to re-run:
    it skips chunks already embedded at this version and commits progress
    so an interrupted job resumes instead of starting over."""
    done = checkpoint.load(new_version.namespace)   # set of chunk ids already indexed
    batch = []

    for chunk in source_chunks:
        if chunk.id in done:
            continue
        batch.append(chunk)
        if len(batch) < batch_size:
            continue
        _flush(batch, new_version, checkpoint)
        batch.clear()

    if batch:
        _flush(batch, new_version, checkpoint)


def _flush(batch, version: EmbeddingVersion, checkpoint):
    texts = [c.text for c in batch]
    vectors = embedding_client.embed_batch(texts, model=version.model)
    vector_store.upsert(
        namespace=version.namespace,
        items=[
            # id is content-stable, so re-running upserts in place, never dupes.
            {"id": c.id, "values": v, "metadata": {**c.metadata, "v": version.model}}
            for c, v in zip(batch, vectors)
        ],
    )
    checkpoint.commit(version.namespace, [c.id for c in batch])

Two things earn their place here. Batching the embedding calls is where your cost and wall-clock actually live; a per-chunk loop against the API will be slower and pricier by an order of magnitude. And the content-stable id means an upsert is idempotent, so a retried batch overwrites rather than duplicates, which matters because a run this long will retry.

While this job runs, production keeps querying the old namespace. Users see nothing. You are building a parallel index next to the live one, not mutating the live one in place.

Rule three: do not cut over on vibes, cut over on recall

This is the step teams skip, and it is the step that catches the disaster. Before you send a single user query to the new namespace, prove it retrieves at least as well as the old one on a labeled set. You need queries paired with the chunk ids that should come back for them. Build this set once and it pays for itself on every future migration, which there will be more of.

def evaluate(queries, version: EmbeddingVersion, k=10):
    """queries: list of (text, relevant_ids:set). Returns recall@k and MRR@k
    for one namespace, so two versions can be compared head to head."""
    recall_sum = 0.0
    rr_sum = 0.0

    for text, relevant in queries:
        qv = embed_query(text, version)
        hits = vector_store.query(namespace=version.namespace, vector=qv, top_k=k)
        retrieved = [h.id for h in hits]

        found = set(retrieved) & relevant
        recall_sum += len(found) / len(relevant)      # how much of the truth we got

        rank = next((i for i, cid in enumerate(retrieved, 1) if cid in relevant), None)
        rr_sum += (1.0 / rank) if rank else 0.0        # how high the first hit ranked

    n = len(queries)
    return {"recall_at_k": recall_sum / n, "mrr_at_k": rr_sum / n}


def cutover_allowed(old_v, new_v, queries, k=10, tolerance=0.01) -> bool:
    old = evaluate(queries, old_v, k)
    new = evaluate(queries, new_v, k)
    print(f"old: {old}  new: {new}")
    # Require the new index to match or beat the old within a small tolerance
    # on BOTH metrics. A higher benchmark score is not permission to regress
    # on your own data.
    return (
        new["recall_at_k"] >= old["recall_at_k"] - tolerance
        and new["mrr_at_k"] >= old["mrr_at_k"] - tolerance
    )

Run this in CI on every migration and you turn a silent quality regression into a red build, the same way LLM evals in CI catch generation regressions before your users do. The public benchmark told you the new model is better on average. This tells you whether it is better on your corpus and your queries, which is the only comparison that pays your bills. I have seen a model that wins on MTEB lose on a narrow domain corpus because the domain vocabulary was underrepresented in its training. The eval gate is what stands between that model and your users.

Rule four: cut over with a pointer, so rollback is instant

When the gate passes, do not rewrite application code to point at the new namespace and redeploy. Put a single indirection between your app and the physical namespace, flip it, and keep the old namespace warm. If retrieval quality or latency looks wrong in the first hour of real traffic, you roll back by flipping the pointer, not by running the entire reindex again in reverse.

# A tiny alias table (a row in your DB or a KV entry) decides which physical
# namespace "live" points at. The app always reads through this.
def active_version(alias_store) -> EmbeddingVersion:
    return alias_store.get("rag:live")

def promote(alias_store, new_version: EmbeddingVersion):
    alias_store.set("rag:previous", alias_store.get("rag:live"))
    alias_store.set("rag:live", new_version)   # atomic flip; next query uses it

def rollback(alias_store):
    prev = alias_store.get("rag:previous")
    if prev:
        alias_store.set("rag:live", prev)      # seconds, not hours

Keep the previous namespace for a defined window, a week is reasonable, then delete it to stop paying double storage. Deleting it is the last step of the migration, not part of the cutover, because the whole point of building the second index instead of mutating the first is that going back has to be cheap.

Tradeoffs and pitfalls

A new dimension or distance metric is a harder break, not a smaller one. If the new model outputs 3,072 dimensions where the old gave 1,536, or expects dot product where you indexed on cosine, the vectors cannot share an index at all and some stores need the index recreated from scratch. The version tag catches this early, which is exactly why the dimension and metric live in it.

Do not change chunking and embeddings in the same migration. If you resize your chunks or rewrite your splitter at the same time you swap the model, and retrieval moves, you will not know which change moved it. Migrate one variable at a time so the eval gate attributes the result to a cause.

Embedding drift is not only an upgrade event. Even without changing models, new content, new terminology, and shifting query patterns pull your effective retrieval quality away from where it was the day you launched. Keep the labeled eval set and run it on a schedule, not just during migrations, so you catch the slow rot as well as the sharp break.

A passing eval is necessary, not sufficient. Retrieval getting the right chunks does not guarantee the model uses them faithfully, so keep your grounding and faithfulness checks on the generation step running after the cutover. The migration gate protects retrieval. It does not replace the guardrail on the answer.

The takeaway

Treat every embedding model change as a versioned data migration: tag vectors with the model that made them, backfill into a fresh namespace while the old one keeps serving, gate the cutover on a labeled retrieval eval, and flip a pointer you can flip back. The reason this matters is that the failure mode is invisible. You do not get an exception, you get worse answers, and worse answers do not page anyone until a customer notices.

If you are sitting on a RAG system whose embedding model has quietly fallen a generation behind because nobody wanted to risk the swap, book a consultation call and we can map a migration path that proves the new index is better before it ever serves a user.