Your LLM Provider Is Slow, Not Down, and the Breaker Never Trips

Early last month a team I was helping watched their agent's p95 latency walk from about four seconds to nine over the course of an afternoon. No alarms. The error rate was flat at its usual fraction of a percent, the model provider's status page was all green, and the circuit breaker they had built after the last outage sat closed and quiet the entire time. By the time anyone looked, the request queue was deep enough that even fast calls were waiting behind slow ones, and a feature that normally felt instant felt broken.

The provider had not gone down. It had gone slow. Calls that usually returned in under a second were taking five or six, still well inside the ten second timeout, so every one of them counted as a success. The breaker counts failures, and there were none. The thing that was hurting them had no error code attached to it, so none of the machinery they had built to survive a provider going dark could see it.

This is the failure mode the outage playbook misses. Everyone builds for the provider that returns 500s or stops answering, because that is the dramatic one and it trips every detector you have. The quieter and more common event is the brownout: the provider still answers, still returns 200, just takes multiples of its normal time, and sits in the blind spot between "working" and "timed out" where none of your reliability logic is looking.

The failure: a provider that is up but slow

Start with what a brownout actually looks like on the wire. Every request completes. Every response is a valid 200 with a well-formed body. If you tail your logs you see a clean stream of successful calls. The only thing that has changed is the number in the duration field, and it has changed for enough of the calls that your aggregate latency moves, but not for all of them, so the median can still look almost normal while the tail blows out.

That last point is why the median lies to you. A provider under load rarely slows every request evenly; it slows a growing fraction of them. Your p50 drifts up a little, your p95 and p99 climb hard, and because an agent turn often makes several model calls in sequence, the turn's latency is closer to the tail of the distribution than the median. A turn that calls the model six times only needs one of those calls to land in the slow fraction to feel slow, so the user-visible slowness is worse than any single-call average suggests. This is the same reason tail latency dominates once you fan out across many calls, the point Dean and Barroso made about interactive services: the more calls a request depends on, the more its latency is governed by the slowest few.

The operational trap is that the brownout grows slowly and the dashboards you built accumulate rather than ring. An outage is a cliff that pages instantly. A brownout is a ramp: latency creeps, the queue deepens, timeouts start firing an hour in once the slow calls finally cross the timeout line, and only then does anything that looks like a failure show up. By that point the cause and the symptom are an hour apart, and you are debugging a queue collapse instead of a slow provider.

Why your circuit breaker sleeps through a brownout

The circuit breaker you have is almost certainly an error-rate breaker. It watches a rolling window of calls, counts the ones that threw or returned a failure status, and opens when that count crosses a threshold. It is the right tool for a provider that errors, and it is the backbone of the failover gateway you build for a hard outage. During a brownout it is deaf, because its entire input signal is the failure count, and the failure count is zero.

This is a textbook instance of what the Microsoft Research team named gray failure: the dependency is degraded in a way the failure detector cannot observe. Their phrase for it is differential observability: the application experiences the problem acutely while the system's own health checks report fine, because the checks measure the wrong dimension. A liveness probe that asks "did the call return" gets a yes. The question that matters, "did it return in time," was never asked.

The timeout does not save you either, and it is worth seeing why. A timeout is a single hard line: below it, success; above it, failure. Brownouts live below the line on purpose. If your normal latency is 800 milliseconds and your timeout is ten seconds, a call that takes six seconds is a seven-fold slowdown that your system records as a perfectly good success. You could lower the timeout, but a tight timeout turns every latency spike into a wave of hard failures and retries, which is its own outage. The timeout is a backstop for calls that hang, not an instrument for calls that are merely slow. You need something that reads the gradient between fast and timed out, and treats "too slow, too often" as a state worth acting on.

Measure slow, not just failed

Before you can trip on slowness you have to measure it, per provider, continuously, as a rate rather than an average. The unit that matters is the slow-call rate: of the calls in the recent window, what fraction took longer than a threshold you consider healthy. Averages hide the tail; a rate of calls over a line exposes it. Pick the threshold from the provider's own normal behavior, something like two or three times its typical p95, not from the timeout.

import time
from collections import deque

class LatencyWindow:
    """Rolling view of recent calls for one provider. Reports the
    fraction that ran slower than `slow_ms`, the signal a brownout moves."""
    def __init__(self, slow_ms: float, window: float = 30.0):
        self.slow_ms = slow_ms
        self.window = window            # seconds of history to keep
        self.calls: deque[tuple[float, float]] = deque()  # (finished_at, duration_ms)

    def record(self, duration_ms: float):
        now = time.monotonic()
        self.calls.append((now, duration_ms))
        cutoff = now - self.window
        while self.calls and self.calls[0][0] < cutoff:
            self.calls.popleft()

    def slow_rate(self) -> float:
        if len(self.calls) < 20:        # not enough data to judge
            return 0.0
        slow = sum(1 for _, d in self.calls if d >= self.slow_ms)
        return slow / len(self.calls)

Two details keep this honest. The minimum sample count stops a single slow call in a quiet window from reading as a 100 percent slow rate and tripping everything. And the window is time-bounded, not count-bounded, so a provider that has gone slow and is therefore serving fewer calls per second still gets judged on recent reality rather than on a stale count of old fast calls. Feed this the same duration you already emit to your traces, so the signal lines up with the latency you see in OpenTelemetry GenAI spans and you are measuring the thing your users feel, not a proxy for it.

A slow-call breaker and early failover

With a slow-call rate in hand, the breaker gets a second trip condition. It still opens on errors, because a provider can do both. It now also opens when the slow-call rate crosses a threshold over a full window, which is the brownout signal. This is not exotic: it is the slow-call rate threshold that Resilience4j ships in its standard circuit breaker, where slowCallRateThreshold and slowCallDurationThreshold open the circuit when too many calls run long, alongside the ordinary failure-rate path. The idea is mature in general backend engineering; model gateways just rarely wire it up.

def should_open(win: LatencyWindow, error_rate: float,
                slow_limit: float = 0.5, error_limit: float = 0.2) -> str | None:
    """Open on either signal. A brownout trips the slow path; a hard
    outage trips the error path. Return which, or None to stay closed."""
    if error_rate >= error_limit:
        return "error_rate"
    if win.slow_rate() >= slow_limit:        # half the window ran slow
        return "slow_rate"
    return None

# On open, route new calls to the next healthy provider instead of
# waiting out the slow primary. Keep the primary in half-open probe mode:
# send it a trickle, watch its slow_rate, fail back only when the tail
# has recovered for a sustained period, not on the first fast probe.

The response to a slow-rate trip is deliberately softer than to an error trip. A provider that is erroring is useless and you cut it off hard. A provider that is slow still works, so failing over trades a known-slow path for a fallback that may format differently or be weaker, the same quality risk you manage in any multi-provider failover chain. Keep the model pinned per provider, validate every response against one schema regardless of which backend served it, and run a small eval against the fallback on a schedule so you are not meeting its quirks for the first time mid-incident. Then fail back slowly: a brownout often recovers in waves, and flapping between providers on every wobble is worse than sitting on one slightly slow path.

One more guard belongs here. Gate the failover behind a sustained window so a brief spike does not swing your whole fleet onto the fallback at once, which can brown the fallback out too. This is the same discipline as shedding load with backpressure instead of piling it on: the goal is to move deliberately under stress, not to thrash.

Hedge the tail, but bound it

Failover is the right move when a provider is broadly slow. For the narrower case of a healthy provider with an unlucky slow tail, where most calls are fast and a few stragglers ruin specific turns, the sharper tool is the hedged request. Send the call, and if it has not returned after a short delay tied to your normal latency, send a second copy to the same or a different provider and take whichever answers first. Dean and Barroso's tail-at-scale work is the canonical source for this: a small amount of redundant work cuts the tail dramatically because the probability that two independent calls are both slow is much lower than one being slow.

import asyncio

async def hedged_call(call, providers, hedge_after_ms=1200, cap=0.1):
    """Fire the primary; if it is slow, fire one hedge. Take the first
    result, cancel the loser. `cap` bounds how often hedging is allowed
    so a brownout cannot double total load at the worst moment."""
    tasks = [asyncio.create_task(call(providers[0]))]
    try:
        done, _ = await asyncio.wait(tasks, timeout=hedge_after_ms / 1000)
        if done:
            return done.pop().result()          # primary beat the hedge delay
        if allow_hedge(cap):                     # only if under the hedge budget
            tasks.append(asyncio.create_task(call(providers[1])))
        done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
        for t in pending:
            t.cancel()                           # free the loser's slot and spend
        return done.pop().result()
    finally:
        for t in tasks:
            if not t.done():
                t.cancel()

The cap is the part people skip and regret. Hedging is nearly free when the tail is thin, because the hedge rarely fires. During a brownout the tail is not thin: a large fraction of calls are slow, so a large fraction trigger a hedge, and an uncapped hedge policy doubles your request volume exactly when the provider can least handle it, turning a brownout into an overload. Bound the hedge rate to a few percent of traffic, and cancel the loser so you are not paying for two completions. Cancellation has to actually propagate to the provider, or you free the local slot while the abandoned call keeps running and billing, the same leak as a deadline that is not propagated through the call chain.

The parts that bite

A slow threshold set from the median trips on normal variance. If you define "slow" as anything above your p50, half your calls are slow by definition and the breaker lives open. Set the slow-call duration from a healthy p95 with headroom, so only a real shift in the distribution crosses it. Measure what normal actually looks like per provider before you pick the number.

Shared state matters more here than for outages. An error-rate breaker on a dead provider converges fast across replicas because every replica sees the errors immediately. A brownout is subtler, and if each replica keeps its own latency window in local memory, they disagree about whether the provider is slow and flap independently. Beyond a single process, the slow-call counts belong in a shared store so the fleet makes one decision.

Hedging without a budget is how you cause the next incident. It is the most satisfying fix and the most dangerous one. The math that makes hedging cheap assumes a thin tail, and a brownout violates that assumption precisely when you reach for it. Cap the hedge rate, cancel losers, and watch hedge volume as its own metric so you can see it climbing before it bites.

Failover can carry the brownout with it. If the primary is slow because the whole region or a shared upstream is stressed, your fallback may be feeling the same pressure, and dumping all traffic onto it at once finishes the job. Move traffic in steps, keep the primary in half-open probe mode, and treat a fallback that also browns out as a signal to shed load rather than to keep searching for a fast provider that does not exist right now.

Quality drift hides inside a latency fix. Because a brownout failover swaps models for a performance reason, not a correctness one, it is easy to forget the output changed. A fallback that is 300 milliseconds faster but wrong more often is not a win. Keep the schema validation and the fallback eval in the loop so a latency decision never silently becomes a quality regression.

The takeaway

The provider was never down. It was slow, in the exact band between healthy and timed out that your outage machinery does not watch, and because every call still returned 200, the breaker you built for the last incident sat closed while latency doubled and the queue backed up behind it. The gap was not a missing failover path. It was a missing signal: nothing in the system was measuring slowness as a condition worth acting on, so nothing acted.

Closing the gap is cheap and mostly about instrumentation. Measure the slow-call rate per provider on a time window, trip the breaker on slowness as well as errors, fail over deliberately when a provider stays slow, and hedge the thin tail under a hard budget so the cure does not become the disease. A brownout then turns from a mystery afternoon into a state your gateway names and handles on its own. If your latency keeps creeping up every few weeks and nothing ever pages, book a consultation call and we can find the brownout your breaker cannot see and give it something that can.