Your Failover Provider Shares the Cloud That Just Went Down

Last week four platforms a lot of AI stacks quietly depend on degraded inside the same window, and plenty of teams with a tidy two-provider failover took the outage anyway. Counting vendors is not measuring resilience.

Viral RuparelNo. 94 of 9411 min read

A team I was helping ran their agent on two model providers, a primary and a fallback, with a clean gateway between them. They had read the postmortems, they had done the responsible thing, two vendors instead of one. Last week, inside a ten minute window, both providers went quiet at once. Their dashboard was almost reassuring: the circuit breaker tripped on the primary exactly as designed, the gateway failed over, and then the fallback timed out too. They had built failover and still took the outage, because both providers served their traffic from the same cloud region, and the cloud region was the thing that was actually down. The failover worked perfectly. It just had nowhere healthy to go.

That week was not a one-off. Over five days in the middle of November, four platforms that a large share of AI stacks quietly sit on degraded in overlapping windows. Each incident looked contained on its own. Together they made a point that a vendor count cannot: resilience is not how many providers you have, it is how many independent ways your providers can fail. If your primary and your fallback go down for the same reason, you do not have two providers. You have one, billed twice.

Two providers drawn as separate boxes that both route down into one shared cloud region, so a single outage at the bottom takes out both paths, next to a fixed layout where the fallback sits in a different failure domain.
Figure. Correlated failover versus independent failure domains

The failover you built assumes an independence you never checked

Most resilience work stops at the gateway. You put a failover gateway in front of your model calls, with a circuit breaker per provider and a hard attempt cap, and that layer is genuinely necessary. But it quietly assumes something it never verifies: that when the primary fails, the fallback is healthy. The whole value of failover is the independence of the two paths, and independence is the one property the gateway code cannot see. It knows the provider names. It does not know that both names resolve to the same data center.

This is why concentration risk is so easy to ship. The design review asks "what if the provider goes down?" and the answer, "we fail over to a second provider," sounds complete. Nobody asks the follow-up: "down for what reason, and does that reason also take out the second one?" Two vendors on the same cloud fail together when the cloud has a bad day. Two vendors serving the same open-weights model from the same inference backbone fail together when that backbone chokes. Two vendors behind the same CDN fail together when the edge has a routing incident. In each case you passed the design review and bought nothing.

The Cloud Security Alliance put the November cluster plainly in their research note on correlated outages: model diversity did not buy resilience at the data layer, and concentration should be measured by shared dependency, not vendor count. That sentence is the whole essay. The rest of this is how to actually do it.

Where the shared dependency hides

Start by refusing the vendor-name abstraction and looking at the stack each request really traverses. Two providers can diverge at the top and converge lower down, and it is the lowest shared layer that decides whether they fail together.

The cloud region is the usual culprit. A surprising number of model providers are tenants on the same few hyperscaler regions, and a regional control-plane or networking incident takes out every tenant at once regardless of their logos. The serving backbone is the next one, and it is subtler: with the same open-weights models now cross-hosted across many inference platforms, your "two providers" may be two frontends in front of one physical fleet, or two platforms that both autoscale on the same scarce accelerator pool. The edge is the one people forget, because a CDN or DNS provider that fronts both of your providers is a shared dependency even when the compute behind them is fully separate. And then there is the control plane every path shares: a single identity provider, a single secrets store, a single config service that, when it stutters, makes every healthy backend unreachable at the same instant.

The uncomfortable version of this: the shared dependency is often not at your providers at all, it is in you. If your gateway, your auth, your rate limiter, or your config loader is a single instance, then every provider in the world is downstream of your own single point of failure, and no amount of vendor diversity helps. Map the whole path, not just the half that has vendor names on it.

Map your failure domains, do not count your vendors

Make the hidden thing explicit. For every path in your failover chain, declare where it actually runs, not what it is called. A short manifest you keep next to the code is enough, and the act of filling it in is where most of the surprises surface.

# providers.yaml: declare where each path really runs, not just its name
providers:
  - name: primary-vendor-a
    model: vendor-a/large
    cloud: aws
    region: us-east-1
    serving: vendor-a-managed     # who runs the inference fleet
    edge: cloudfront
  - name: fallback-vendor-b
    model: vendor-b/large
    cloud: aws                     # same cloud as primary
    region: us-east-1              # and the same region: shared domain
    serving: vendor-b-managed
    edge: cloudfront               # and the same edge
  - name: fallback-self-hosted
    model: vendor-c/open-weights
    cloud: gcp                     # different cloud, different region
    region: europe-west4
    serving: self                  # you run it: a different failure mode
    edge: none

Written out like this, the problem is obvious: the first two "providers" share cloud, region, and edge, and differ only in the vendor name, which is the one field that does not cause outages. Now turn the manifest into a check that fails loud, so a fallback that shares a failure domain with the primary cannot quietly ship.

SHARED_DOMAINS = ("cloud", "region", "serving", "edge")

def failure_domains(p):
    # the things that, if they break, take this path down with them
    return {d: p[d] for d in SHARED_DOMAINS if p.get(d) not in (None, "none")}

def independent(a, b):
    da, db = failure_domains(a), failure_domains(b)
    shared = {d: da[d] for d in da if db.get(d) == da[d]}
    return (not shared, shared)

ok, shared = independent(primary, fallback)
if not ok:
    # failover between these two is theater: they fail in the same domain
    raise SystemExit(f"fallback shares {list(shared)} with primary: {shared}")

This is deliberately crude, and that is the point. It encodes the one rule the design review keeps skipping: a fallback only counts if it shares no failure domain with the thing it is backing up. Run it in CI against your providers.yaml and the day someone adds a convenient second vendor that happens to live in the same region, the build tells them it bought nothing.

Measure correlation from your own telemetry

The manifest captures what you believe. Your logs capture what is true. Declared independence drifts: a provider migrates regions, a vendor silently moves inference to a cheaper backbone, a cross-hosted model lands on the same fleet as your primary. The only way to catch that is to measure how often your providers are failing in the same minutes, using the request outcomes you already record.

Bucket your calls by the minute, per provider, and ask the blunt question: when the primary is having a bad minute, how often is the fallback also having one? That conditional rate, not the raw error rate of either provider alone, is the number that tells you whether your failover will work.

from collections import defaultdict

def minute_buckets(events):
    # events: iterable of (provider, ts_seconds, is_error)
    buckets = defaultdict(lambda: defaultdict(lambda: [0, 0]))
    for provider, ts, is_error in events:
        errs_total = buckets[int(ts // 60)][provider]
        errs_total[1] += 1
        errs_total[0] += 1 if is_error else 0
    return buckets

def is_bad(errs_total, threshold=0.2):
    errs, total = errs_total
    return total > 0 and errs / total >= threshold

def co_failure_rate(buckets, primary, fallback):
    primary_bad = both_bad = 0
    for by_provider in buckets.values():
        if primary in by_provider and fallback in by_provider:
            if is_bad(by_provider[primary]):
                primary_bad += 1
                both_bad += 1 if is_bad(by_provider[fallback]) else 0
    # P(fallback bad | primary bad): high means it is not a real fallback
    return both_bad / primary_bad if primary_bad else 0.0

A co-failure rate near zero means your fallback is genuinely holding up the minutes your primary struggles, which is exactly what you paid for. A rate near one means the two move together, and the second provider is decoration. Track this as a first-class reliability metric, next to the error budget you already spend against an SLO, and alert when it climbs, because a rising co-failure rate is an early warning that two paths you believe are independent have quietly converged. This is the same discipline as a breaker that trips on slowness rather than only on errors: you measure the failure mode that actually hurts, not the one that is easy to see.

Pick fallbacks for independence, then keep proving it

Once you can see correlation, choosing a fallback stops being about picking the second-best model and becomes about picking the most independent failure domain you can afford. A fallback on a different cloud, served by a different operator, behind a different edge, is worth more as insurance than a slightly stronger model that shares your primary's region. The strongest independence usually comes from changing the kind of thing, not just the brand: a self-hosted open-weights model on your own infrastructure fails for completely different reasons than a managed API, which is also why keeping a second-source model you can migrate to pays off twice, once for deprecation and once for correlated outages. This does not fight your cost work either: cost-aware routing still picks the cheapest capable model on a normal day, and the independence constraint only governs what sits underneath it as the backstop.

Independence rots the moment you stop looking, so prove it on a schedule rather than trusting the manifest. Run a small job that sends a real request down every path and records the outcome, and periodically recompute the co-failure rate over the last few weeks so a provider that migrated into your primary's region shows up as a correlation spike before the next incident does. AWS frames the underlying principle well in its reliability guidance on fault isolation: components should be placed in separate fault isolation boundaries so that a failure in one is contained rather than shared. Your providers are components. The boundary you care about is the lowest layer they have in common.

And decide honestly what happens when the whole class is down, because sometimes it will be. If every provider you can reach lives in regions that a single hyperscaler incident took out, no failover conjures a healthy backend, and your degraded mode is the only thing standing between you and a spinner that lies. A cached answer, a non-generated path, or a clear "the model is unavailable and your work is saved" is what users remember kindly. Independence buys you fewer of these moments. It does not promise none.

The parts that bite

Transitive dependencies hide below your manifest. You can diversify cloud, region, and operator and still sit two different providers on the same tier-one transit network or the same DNS root, and discover the shared layer only when it fails. You cannot enumerate all of it, so focus the effort on the domains that fail often, cloud region and edge and serving backbone, and accept that the bottom of the stack carries residual risk you cannot design away.

Cross-hosting moves correlation under your feet. The same open-weights model now runs on many platforms, which is great for availability until two of your "independent" providers autoscale onto the same scarce accelerator capacity during a demand spike and throttle in lockstep. The cross-hosting of frontier models means model diversity and infrastructure diversity are not the same thing, and only the measured co-failure rate will catch the difference.

Your gateway is the single point of failure you built yourself. All this provider diversity routes through one resilience layer, and if that layer is a single instance, a single config load, or a single auth path, it is the real concentration risk and every provider is downstream of it. Give it the same independence treatment: more than one instance, in more than one zone, with no shared fate at the choke point.

Testing independence is not free, and that is the honest trade. Keeping a genuinely independent fallback warm costs money for a path you hope to never use at full load, and the scheduled probes and the self-hosted backstop add operational weight. The spend is small against a correlated outage that takes your product dark for an afternoon, but it is real, and pretending the second path is free is how it quietly rots into a fallback that fails the day you finally need it.

The takeaway

Two providers is a reassuring number, and it is the wrong number to track. What decides whether you survive the next bad week is how many independent ways your providers can fail, and last week showed that four platforms a lot of stacks trust can wobble inside the same five days. Resilience is measured at the lowest layer your paths share, not at the vendor list, and correlated failure is quiet precisely because every piece of your failover looks like it is working while the request dies anyway.

Write down where each path actually runs, fail the build when a fallback shares a failure domain with the thing it backs up, and measure from your own logs how often your providers are bad in the same minute. Then move your backstop somewhere that fails for different reasons, and keep proving it does. If your failover has never been tested against a correlated outage and you would rather not learn its co-failure rate during the incident, book a consultation call and we can map your failure domains and find the dependency your two providers quietly share.

ShareXLinkedInviralruparel.com/blog/llm-provider-concentration-risk-correlated-failover

Questions this essay answers

01What is LLM provider concentration risk?

It is the risk that two or more of your model providers fail together because they share a dependency you did not account for: the same cloud region, the same inference backbone hosting the same open-weights model, the same CDN or DNS, or the same identity provider. You can have two vendors and one failure domain. Concentration risk is what turns a careful multi-provider setup into a single point of failure, and it does not show up on a vendor count. It shows up the first time both providers go quiet in the same minute.

02How is correlated failure different from a normal provider outage?

A normal outage takes down one provider, and failover to a second one restores service. Correlated failure takes down the second provider at the same time, because both depend on something that broke underneath them. The failover still fires, the breaker still trips, the dashboard still shows the gateway doing its job, and the request fails anyway because there was never a healthy backend to route to. The fix is not better failover logic. It is choosing a fallback that fails for different reasons than the primary.

03How do I know if my two providers share a failure domain?

Two ways, and you want both. First, declare where each provider actually runs: cloud, region, who serves the inference, which edge and DNS sit in front of it, and check that no two paths in your chain share a value. Second, measure it from your own telemetry: bucket your request errors by the minute per provider, and compute how often the fallback is failing in the same minutes the primary is. A high conditional failure rate means the two are not independent, whatever the vendor names say.

04Can I remove concentration risk completely?

No, and chasing zero is the wrong goal. Everything shares something eventually: a tier-one network, a root DNS, a certificate authority. The goal is to push the shared dependency far enough down the stack that it almost never fails, and to make sure your primary and fallback do not share anything higher up that fails routinely. Measure the correlation, keep it low for the domains that break often, and accept the residual risk at the bottom of the stack that you cannot engineer away.

Working on this in production?I help teams make agents like this reliable. Thirty minutes, no pitch.
Viral Ruparel

Generative AI consultant and full-stack architect. Ex-IBM, former Director of Engineering at a YC W20 startup. Writes about what breaks when agents meet reality.