Use a Circuit Breaker When Retries Multiply an Outage

The retry cap that does not do what it looks like

Your team runs evals through a small internal gateway that sits in front of a model provider. Each eval job retries a failed call at most twice. That looks careful: it bounds each job's cost, latency and worst case. Then the provider starts returning 529 overloaded_error, and 3,000 jobs are starting. Each one tries, fails, retries, fails, retries, fails, and gives up. That is 9,000 requests aimed at a service that was already overloaded before the first of them arrived.

The per-job cap did what it promised. It bounded each job. It could not stop thousands of jobs from hammering a service that was already down, because no job can see the others. This article derives how retries multiply across layers and why the multiplier is largest during an outage. It then simulates the effect and builds the pattern that addresses it: a shared circuit breaker.

The book's Chapter 12 derives what retries do to one request's side effects. Chapter 19 covers what a model call costs, batching, and where the latency tail comes from. What neither covers is the fleet-level arithmetic, which is what this article adds.

Retries multiply across layers

A call from an eval job often passes through several layers that each retry: your harness, a gateway, and the provider's SDK. Anthropic's SDKs, for example, retry transient failures twice by default, with exponential backoff, and expose max_retries to change that (Claude API errors).

Suppose each of d layers makes one attempt plus up to r retries, and the dependency is fully down, so every attempt fails. The innermost layer sends r + 1 times. The layer above calls it r + 1 times, and so on. One logical request becomes

sends = (r + 1)^d

Three layers with three retries each give 4³ = 64 sends. With a harness and gateway at three retries and the SDK at its default two, you get 4 × 4 × 3 = 48. The Google SRE book names the problem: "If multiple layers retried, we'd have a combinatorial explosion" (Site Reliability Engineering, "Handling Overload").

Why the multiplier peaks in an outage

Now suppose the dependency fails each attempt independently with probability f. The innermost layer sends 1 + f + f² + … + fʳ times on average, and it fails only if every attempt failed, which happens with probability f^(r+1). That failure probability is what the next layer sees. Applying this layer by layer gives the expected sends per logical request. For three layers of three retries:

At moderate failure rates, inner retries hide most failures from outer layers, and the multiplier stays near 1 or 2. As the dependency approaches total failure, every layer's retries fire and the multiplier climbs toward 64. Retries add the most load exactly when the dependency can least absorb it.

It gets worse if the dependency's failure rate rises with its load, as an overloaded server's does. Then retries raise the load, the higher load raises f, and a higher f raises the multiplier. That feedback loop can keep a service down after the original cause is gone. This is the sense in which retries make an outage worse.

The function below computes the table, so you can put in your own layers. It runs on Colab's Python 3.12.

def amplification(f, retries):
    """Expected sends to the dependency per logical request.

    retries lists the retries allowed at each layer, innermost first.
    The dependency fails each attempt independently with probability f.
    """
    sends, fail = 1.0, f
    for r in retries:
        attempts = sum(fail**i for i in range(r + 1))  # calls into the layer below
        sends *= attempts
        fail = fail ** (r + 1)  # this layer fails only if every attempt failed
    return sends


for f in (0.1, 0.5, 0.9, 0.99, 1.0):
    print(f, round(amplification(f, [3, 3, 3]), 2), round(amplification(f, [2, 3, 3]), 2))

The second column is the table above. The third puts an SDK's default two retries innermost: 48 at f = 1.

Two fixes: one layer retries, and a budget caps it

The SRE book describes two controls. First, a per-request limit: "If a request has already failed three times, we let the failure bubble up to the caller." Second, a per-client retry budget: "A request will only be retried as long as this ratio is below 10%" (Handling Overload). With a 10% budget, a layer can add at most 10% to its load, whatever f is, so the multiplier per layer is capped at 1.1. It also advises that only the layer immediately above a failing component should retry it.

In practice: pick one layer to own retries, set max_retries=0 or a small number in the others, and count sends at the boundary. AWS's Well-Architected guidance says the same: bound retries, use backoff and jitter, and do not multiply attempts across layers (AWS Well-Architected: control and limit retries).

Budgets and single-layer retries bound the multiplier per request. They still let every job send its first attempt into a dependency that is down. Stopping those first attempts needs memory shared across jobs.

Three states, one shared memory

Martin Fowler describes the circuit breaker: you wrap a protected call in an object that monitors for failures, and "once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all." After a suitable interval, a call in the half-open state "results in a trial call, which will either reset the breaker if successful or restart the timeout if not" (Fowler, "CircuitBreaker").

• Closed: calls pass through; the breaker counts recent failures.
• Open: after the threshold, calls are refused locally and at once. This is what a per-job counter cannot do.
• Half-open: after the cooldown, one probe is admitted. Success closes the breaker; failure reopens it and restarts the cooldown.

Key the breaker to the dependency, not the job, and share it across every worker that calls that dependency: in one process, a lock; across a fleet, a shared store or the gateway itself. Probe admission must be atomic, so that when the cooldown ends, exactly one caller gets through instead of every waiting worker at once.

View full-size diagram

Simulate it

The simulation below sends 3,000 jobs at five per second into a dependency that is down for the first 300 seconds. Each job retries twice, after 1 and 2 seconds. With a breaker (three failures open it, 30-second cooldown, one probe), a refused job is parked and checked again 30 seconds later instead of being dropped. It uses a fake clock and no network, so it is deterministic.

import heapq


class Breaker:
    def __init__(self, threshold=3, cooldown=30.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.state, self.failures, self.opened_at, self.probe_out = "closed", 0, None, False

    def admit(self, now):
        if self.state == "open" and now >= self.opened_at + self.cooldown:
            self.state = "half_open"
        if self.state == "closed":
            return True
        if self.state == "half_open" and not self.probe_out:
            self.probe_out = True
            return True
        return False

    def record(self, now, ok):
        if ok:
            self.state, self.failures, self.probe_out = "closed", 0, False
            return
        self.failures += 1
        if self.state == "half_open" or self.failures >= self.threshold:
            self.state, self.opened_at, self.probe_out = "open", now, False


def simulate(use_breaker, jobs=3000, rate=5.0, outage_end=300.0, retries=2, backoff=(1.0, 2.0)):
    """Jobs arrive at `rate` per second; the dependency is down until outage_end."""
    breaker = Breaker() if use_breaker else None
    events = [(i / rate, i, 0) for i in range(jobs)]  # (time, job, attempt)
    heapq.heapify(events)
    failed_sends = ok_sends = deferred = gave_up = 0
    while events:
        now, job, attempt = heapq.heappop(events)
        if breaker and not breaker.admit(now):
            deferred += 1  # refused locally; the job is parked and checked again later
            heapq.heappush(events, (now + 30.0, job, attempt))
            continue
        ok = now >= outage_end
        if breaker:
            breaker.record(now, ok)
        if ok:
            ok_sends += 1
        else:
            failed_sends += 1
            if attempt < retries:
                heapq.heappush(events, (now + backoff[attempt], job, attempt + 1))
            else:
                gave_up += 1
    return {"sends_during_outage": failed_sends, "successful_sends": ok_sends,
            "jobs_given_up": gave_up, "local_refusals": deferred}


print("no breaker", simulate(False))
print("breaker   ", simulate(True))

It prints:

no breaker {'sends_during_outage': 4480, 'successful_sends': 1515, 'jobs_given_up': 1485, 'local_refusals': 0}
breaker    {'sends_during_outage': 12, 'successful_sends': 3000, 'jobs_given_up': 0, 'local_refusals': 8270}

Without the breaker, 4,480 requests hit the down dependency, and 1,485 of the 1,500 jobs that arrived during the outage gave up. With it, the dependency saw 12 requests during the outage: three failures to open the breaker, then one probe per cooldown. The 8,270 refusals happened locally, at no cost to the dependency, and every job finished after recovery.

This is a model, and it leaves out two things that matter. The dependency here recovers at a fixed time whatever its load, so the feedback loop above cannot happen; in a real overload, the 4,480 extra sends could delay recovery itself. And the model has no capacity limit, so it hides a new problem the breaker creates. Parked jobs all come back on 30-second boundaries, so recovery brings a burst. Add jitter to the deferral and a concurrency limit on the closed breaker before you trust it.

What a refused job should receive

A refusal needs a response the caller can act on. For an eval job, a reasonable shape is a code such as dependency_temporarily_unavailable, an operation_id for tracing, a safe_action of defer_item, a prohibited_action of score_item_as_failed, and a next_status_check_after_seconds of 30. The prohibited action is the one that matters for research: an eval that scores refused items as wrong answers measures the outage, not the model. The companion article on recovery policies covers keeping infrastructure failures out of scores.

Decide what counts toward the breaker. A 529 or a run of timeouts should. A 401 or a spend-cap 429 should not: retrying will never fix them, and a breaker that opens on them wastes its cooldown on a problem it cannot solve.

Check the state machine with a fake clock

Use a threshold of three failures and a 30-second cooldown. Failures at seconds 0, 1 and 2 open the breaker until second 32.

View full-size diagram

The fixture proves the state machine. It does not show that three failures and 30 seconds suit your dependency. Choose those from the dependency's measured failure and recovery behavior.

Check the decision

What would prove this wrong

If a healthy dependency keeps getting marked unhealthy by brief blips, the threshold is too low or the breaker counts failures it should not, such as ordinary timeouts under normal load. If recovery still brings a burst, check when it starts. Several probes at once point to an admission bug. A burst after the successful probe means parked work resumed faster than the dependency could absorb it. One successful probe does not prove capacity for the whole backlog. Both cases can be scripted in the fixture.

Why this matters in a lab

Labs run large numbers of jobs against a few shared services: an inference gateway, a provider API, a vector store, a results database. When one of those degrades, every eval, sweep and agent job hits it at once. A team whose harness multiplies retries and scores refusals as failures can turn a ten-minute provider incident into a lost night of evals, a larger bill, and wrong numbers.

An interviewer might ask: "Our eval harness, gateway and SDK each retry three times. What happens during a provider outage, and what would you change?" A strong answer derives (r + 1)^d = 64 and explains why the multiplier peaks as f approaches 1. It proposes one retrying layer with a retry budget, adds a shared breaker keyed to the dependency, defers refused items instead of scoring them, and says what the simulation leaves out. For practice, rerun the simulation with the dependency's failure probability rising with the number of requests in flight, and compare recovery times with and without the breaker.