Your LLM Gateway Is One Bad Release From Leaking Every Key

You run a dozen agents and they all talk to the model through one place. Maybe it is a LiteLLM proxy, maybe a gateway you stood up yourself, but the shape is the same: a single service that holds the OpenAI key, the Anthropic key, the Bedrock role, the per-tenant credential table, and routes every call through one process so the rest of your stack does not have to care which provider answered. It is the clean design. It is also the fattest target in your infrastructure, and on 21 October 2026 a lot of teams found out why.

That day a trojanized LiteLLM 1.107.0 went up on PyPI and into the project's official container images. It was live for about two hours before anyone pulled it. According to Microsoft's analysis of the compromise, the release carried a credential stealer that reached cloud keys, SSH and git credentials, AI tool configs, browser databases, and the proxy's own stored provider credentials, encrypted the haul, and posted it to a lookalike domain. The maintainers traced the push to stolen developer tokens, not a flaw in the code. There is no CVE, because nothing was vulnerable. The release was just malicious, and if you installed it, the gateway that held every key you own handed them over.

This is not a LiteLLM story. It is a story about what you concentrated when you put a gateway in the middle, and it will happen again to whatever gateway you run. The fix is not to tear out the proxy. It is to stop treating it as trusted infrastructure and start treating it as the single point where a bad release leaks everything, then engineer for that specific failure.

The gateway is a credential blast radius

Start by naming what the gateway actually is from a security standpoint. It is the one process in your system that, by design, can authenticate to every model provider you use. You gave it the keys on purpose, so that a hundred callers do not each need their own. You probably also gave it a database of per-tenant settings, which for many proxy setups includes stored provider credentials and the connection string to reach them. And it runs on a host with a cloud role, environment variables, and a filesystem full of the ambient secrets every deployment accumulates.

Add that up and the gateway is not holding one secret, it is holding the union of every secret that any path to the model needs. That is the blast radius. When the skill or tool your agent pulls off a registry turns out to be hostile, the damage is scoped to what that one component could reach, which is the whole argument for why you treat every third-party skill as untrusted code and box it in. The gateway inverts that. Its entire job is to reach everything, so a compromise of the gateway is a compromise of everything at once. The convenience and the risk are the same property viewed from two sides.

So the question is not whether you can make the gateway safe to trust. You cannot, any more than you can make any dependency safe to trust by reading its README. The question is how much a compromised gateway can actually take, and how loudly a swap announces itself before it runs.

How a poisoned release runs before you import anything

The reflex when people hear "malicious package" is to assume it only fires if your code imports it. The LiteLLM payload did not wait for that, and understanding why matters, because it changes where you have to look.

The wheel dropped three files into the Python environment's library directory, outside the litellm folder: cook.pth, cook.py, and sitecustomize.py. The first is the one that bites. A .pth file is a path configuration file, and Python has a backward-compatibility behavior that most engineers have never had reason to know about. The CPython site module docs put it plainly: "lines starting with import (followed by space or tab) are executed." So a one-line cook.pth that reads import cook runs cook.py at interpreter startup, in a background thread, before your application imports a single thing. The sitecustomize.py file is the same trick by another door, since site imports that module by name on startup too.

The practical consequence is that any Python process on the host was compromised, not just the gateway. A developer who ran a one-off script, a CI job, a cron task, all of them executed the stealer the moment the interpreter came up. Datadog's write-up is explicit that removing the litellm package does not remove these files, which is why updating alone is not recovery. You detect this by looking for the hooks themselves, not the package.

# Find the startup hooks a poisoned wheel leaves behind.
# They sit in site-packages, OUTSIDE the litellm/ folder, so
# uninstalling or upgrading the package never removes them.
site=$(python -c "import site; print(site.getsitepackages()[0])")
ls -l "$site"/cook.pth "$site"/cook.py "$site"/sitecustomize.py 2>/dev/null \
  && echo "startup hooks present: treat this host as compromised"

# cook.pth is a single line, "import cook", run on every interpreter
# start. Look for any .pth that executes an import, not just this name.
grep -RIl '^import ' "$site"/*.pth 2>/dev/null

Do this from a trusted toolchain on a known-clean system, not by running python or pip inside the suspect environment, because the thing you are investigating runs when that interpreter starts.

Pin the bytes, not the version

The first control is the cheapest and the one that would have caught this. The release was 1.107.0, a version number that looked like a normal point bump. If your deploy pulls "the latest 1.x" or follows a moving tag, a malicious release under a plausible number slides straight in. What you want is to pin the exact bytes you reviewed, so that any change is a loud, reviewable event instead of a silent install.

For Python that means a hashed requirement installed with hashes required. For containers it means a digest, never a tag, because a tag is a label the registry controls and a label can be repointed to a poisoned image under the same name. This is the same reasoning behind recording integrity hashes in a lockfile, and the same instinct that makes you fingerprint an MCP tool's definition and gate on drift rather than trust that a server still does what it did yesterday.

# requirements.txt: pin content, not a floating version.
litellm==1.106.2 \
  --hash=sha256:<the wheel digest you reviewed>

# Install so that any unpinned or mismatched byte fails loudly:
#   pip install --require-hashes -r requirements.txt
# A republished 1.106.2 with different bytes now refuses to install.

# Containers: pin the digest, never a moving tag.
#   FROM ghcr.io/berriai/litellm@sha256:<digest>   # not :latest, not :v1.107.0
# A tag can be repointed to a poisoned image under a name you trust.

The pin lives in your repo under code review, and bumping it is a pull request a human approves after reading the diff, exactly the discipline you already apply to any dependency update. It does nothing about a release that was malicious the day you reviewed it. That is what the next two controls are for, and they assume the pin failed.

Shrink what the gateway can hand over

Assume the poisoned release is running inside your gateway right now. What can it take? Today the answer is usually "the root key for every provider, plus the credential table," because that is what you stored there. You can make the answer much smaller by never giving the gateway a long-lived root key in the first place.

The pattern is a broker. Keep the root provider keys in a separate trust domain, a small service whose only job is to mint short-lived, narrowly scoped tokens, and have the gateway ask it for one per request. The gateway then holds, at any instant, a token good for one tenant and one model for a few minutes, and nothing else. A stealer that dumps the gateway's memory or reads its proxy tables walks away with minutes of one tenant's access instead of your organization's keys. This is the same move as exchanging a request's identity for a scoped, short-lived credential rather than letting a shared principal carry the authority of everyone it ever served.

# key_broker.py -- the gateway never stores a root provider key.
import httpx

async def provider_token(tenant: str, model: str) -> str:
    # The broker holds the root key in its own trust domain and mints
    # a token scoped to one model and tenant, valid for a few minutes.
    async with httpx.AsyncClient(timeout=5) as c:
        r = await c.post(
            "https://broker.internal/mint",
            json={"tenant": tenant, "model": model, "ttl_seconds": 300},
        )
    r.raise_for_status()
    return r.json()["token"]

# The gateway calls this per request and throws the token away after.
# Compromise it and you lose minutes of one tenant's access, not the org.

This does not make the broker free of risk, it moves the concentration to a much smaller, much more auditable surface that does one thing and can be watched closely. You have traded a fat, busy, frequently-updated gateway full of keys for a thin service that mints tokens and changes rarely. That is a far better thing to defend.

Lock the egress so stolen keys have nowhere to go

The last control is the one that holds even when the code is actively hostile and the scan missed it. The stealer did its damage in the final step: it opened a connection to a lookalike domain and posted the encrypted haul out. Socket's breakdown of the artifacts shows the exfiltration target was a domain dressed up to look like the project's own. If the gateway host could not reach arbitrary destinations, that upload fails at the kernel, no matter what the code wanted to do.

So default-deny the gateway's outbound traffic. The proxy needs to reach the provider APIs and your broker, and nothing else. Everything outside that short list is dropped and logged. This is the containment boundary you actually bet on, the same default-deny posture that makes a code-execution sandbox with least privilege worth running, and the same reason you put egress controls in front of what an agent can send.

# Default-deny egress on the gateway host. The proxy reaches the
# provider endpoints and the broker, nothing else. A stealer's POST
# to a lookalike domain dies at the kernel, not at a code review.
nft add table inet egress
nft add chain inet egress out { type filter hook output priority 0 \; policy drop \; }

# Allow only resolved provider and broker IPs on 443, plus DNS.
nft add rule inet egress out ip daddr @allowed_ips tcp dport 443 accept
nft add rule inet egress out udp dport 53 accept

# Everything else is dropped and logged for the on-call to see.
nft add rule inet egress out log prefix "egress-drop " drop

Enforce the allowlist by resolved IP at the host or, better, by hostname at a forward proxy that sees the name before DNS flattens it to an address. Either way the rule runs outside the gateway process where the payload cannot patch it away, and the dropped-packet log becomes the alert that something tried to phone home.

The parts that bite

• Rotation without eviction is theater. If you rotate keys but leave cook.pth and friends on the host, the next interpreter start steals the new keys too. Find and delete the hooks, rebuild from a verified artifact, then rotate from a clean machine, and revoke live sessions because rotation does not always invalidate tokens already issued.
• The pin and an auto-updater cancel out. A hash pin does nothing if your tooling silently bumps to the newest release on deploy. The pin only works when the bump is a reviewed change, so turn off unattended upgrades for anything that holds secrets.
• Egress by IP drifts. Provider endpoints change IPs, so an IP allowlist needs refreshing or it either breaks calls or quietly grows stale and permissive. A hostname-aware proxy is the sturdier version, and it also gives you a clean record for the audit trail that lets you reconstruct what the gateway did after an incident.
• The broker becomes the new crown jewel. You shrank the gateway's blast radius by concentrating the root keys in the broker, so the broker now needs the tightest pinning, the smallest surface, and the slowest change cadence of anything you run. That is the point, a small target is defensible, but do not forget you moved the target.
• Developer laptops are hosts too. The payload ran on anything that started Python with the package installed. A pinned, locked-down production gateway does not help the engineer who ran pip install litellm locally that afternoon. The same hash-pinned lockfile has to govern dev environments, not just the deploy.

The takeaway

The LLM gateway earned its place by concentrating every model call through one door, and that same concentration is why a single poisoned release can empty your entire key store in the two hours before anyone notices. You do not fix that by trusting the dependency harder, because the LiteLLM maintainers did nothing wrong at the code level and it happened anyway. You fix it by engineering for the compromise you now know is possible: pin the exact bytes so a swap is loud, keep root keys out of the gateway so a theft is small, and default-deny egress so a stolen secret has nowhere to go. None of it is exotic, and all of it is the kind of thing teams skip because the gateway felt like plumbing rather than the richest target in the building.

If your agents all route through one gateway or proxy and you are not sure what a bad release of it could reach, book a consultation call and we will map the blast radius and put these three controls in front of it.