Match Each Agent Failure to Its Own Recovery Policy

One retry loop hides the decision that matters

Your eval harness is scoring 5,000 items against a model API overnight. Some calls come back with a rate limit. One comes back saying the organization has hit its spend cap. A few model answers fail your JSON validator. An agent step that writes to the shared results table times out. If all of these go into the same for attempt in range(3) loop, the loop makes the same decision for all of them, and it is wrong for most.

The question to ask of each failure is: what must change before another attempt can succeed? For a rate limit, time must pass. For a spend cap, a person must raise the limit. For a malformed answer, the model needs to see the validator's message. For a timed-out write, you need to find out whether the write happened. Classify first, then decide whether another attempt is permitted and likely to help.

Chapter 12 of the book derives what retries do to side effects and measures it. Chapter 3 shows why an agent loop must stop. This article adds a classification built from a real provider's documented errors, the arithmetic of what each retry buys, and a fault-injection protocol you can run.

Classify from the provider's contract, not the status code alone

Anthropic's API errors page and rate limits page document the cases an eval harness meets most. They show why a status code is not enough on its own.

The two 429 rows are the lesson. Both are rate_limit_error, and a loop keyed on the status code would retry both. One clears in seconds. The other keeps failing until a person acts or the month rolls over, and every retry is wasted work. The documentation tells you how to tell them apart: the spend-cap response has no retry-after header and carries enforced_spend_limit_reached.

The 504 row splits on side effects. A pure model call that times out can be sent again: you may pay for a duplicate generation, but nothing else changes. A tool call that writes a row, launches a job or sends a message is different. "No answer" is not "no effect," and Chapter 12 shows how to reconcile by an operation key before any resend.

What a retry buys, in numbers

Suppose a retry of some failure class succeeds with probability q, independently each time. With at most k retries:

P(success within k retries) = 1 − (1 − q)^k
E[attempts] = 1 + (1 − q) + (1 − q)² + … + (1 − q)^k

For a transient failure with q = 0.5 and k = 3, the success probability is 1 − 0.125 = 0.875, and the expected number of attempts is 1 + 0.5 + 0.25 + 0.125 = 1.875. For a deterministic failure, such as a malformed request or a spend cap, q is close to 0. Then the success probability stays near 0 and the attempts go to k + 1 = 4: four times the cost and latency, for nothing.

Independence is the assumption to distrust. Retries of an overloaded service are correlated, because the overload that failed the first attempt is still there a second later. Backoff and jitter spread attempts out in time. They do not make the failures independent, and that is why the circuit breaker article exists.

Hidden retries multiply

The Anthropic SDKs retry transient failures automatically, twice by default, with exponential backoff, honoring retry-after when present. They expose a max_retries setting to change or disable this (API errors). If your harness wraps each call in its own loop of one attempt plus three retries, each logical call can reach the network up to

(1 + 3) × (1 + 2) = 12 times

That is before any retries in a gateway or proxy you may also run. The general rule, with r retries at each of d layers, is up to (r + 1)^d sends; the circuit breaker article derives it and simulates the load. AWS's Well-Architected guidance on limiting retries says the same for distributed systems in general: bound retries, use backoff and jitter, and do not let retries multiply across layers (AWS Well-Architected, "Control and limit retry calls"). Pick one layer to own retries, set max_retries explicitly in the others, and count sends at the boundary.

View full-size diagram

Build the classifier and the shared budget

The classifier below is a pure function: failure in, recovery action out. It touches no network and no counter. It runs on Colab's Python 3.12.

def classify(failure: dict) -> str:
    """Map one failure to a recovery action. Unknown failures stop."""
    kind = failure.get("kind")
    status = failure.get("status")
    if kind == "schema_error":
        return "repair"  # bounded, with the validator's message fed back
    if kind == "policy_refusal":
        return "refuse"  # a boundary, not a fault
    if status == 429 and failure.get("error_code") == "enforced_spend_limit_reached":
        return "stop"  # the spend cap: retrying fails until access resumes
    if status == 429 and failure.get("retry_after") is not None:
        return "wait_then_retry"
    if status in (500, 529):
        return "backoff_then_retry"
    if status == 504:
        return "reconcile" if failure.get("side_effect") else "backoff_then_retry"
    if status in (400, 401, 403):
        return "stop"  # the same request will fail the same way
    return "unknown_failure"


cases = [
    {"status": 429, "retry_after": 20},
    {"status": 429, "error_code": "enforced_spend_limit_reached"},
    {"status": 529},
    {"status": 504, "side_effect": True},
    {"status": 401},
    {"kind": "schema_error"},
    {"kind": "policy_refusal"},
    {"status": 418},
]
for case in cases:
    print(case, "->", classify(case))

It prints wait_then_retry, stop, backoff_then_retry, reconcile, stop, repair, refuse and unknown_failure, in that order. The last line matters as much as the others. A classifier that sends unrecognized failures to a retry assumes every failure it has not named is safe to loop on. The caller must stop or escalate on unknown_failure; the string alone enforces nothing.

Next, the budget. The loops below reproduce a harness with four attempts wrapped around an SDK with three, where every attempt fails. The shared counter is checked before each send, including the first.

class BudgetExhausted(Exception):
    pass


def nested_attempts(harness_attempts=4, sdk_attempts=3, limit=None):
    """Every attempt fails transiently. Count the sends that reach the transport."""
    sends = 0
    try:
        for _ in range(harness_attempts):
            for _ in range(sdk_attempts):
                if limit is not None and sends >= limit:
                    raise BudgetExhausted
                sends += 1
    except BudgetExhausted:
        return sends, "budget_exhausted"
    return sends, "layers_exhausted"


print(nested_attempts())
print(nested_attempts(limit=4))

It prints (12, 'layers_exhausted') and then (4, 'budget_exhausted'). The counter lives in one process. Workers in separate processes need an atomic reservation in a shared store, so two of them cannot both spend the last attempt. The budget article builds one in SQLite.

View full-size diagram

Add a deadline as a separate limit. Four attempts can still take too long if each one waits for a slow response, and backoff adds its own delay.

A fault-injection protocol you can run

A local test shows the classifier behaves as written. It does not show what reaches the network through your real wrapper stack. To test that, put a stub server between the harness and the provider. The stub returns documented error shapes on a schedule, and it counts every request it receives. You need no API key.

Record the counts in a small table with your harness version, SDK version and settings. That table is the receipt a reviewer can check. This article has not run the protocol against any particular harness; the numbers are yours to produce.

Check the decision

Why this matters in a lab

Long eval runs and agent jobs fail in many small ways overnight. The engineer who has classified those failures gets a clean table in the morning. One who has not gets a table with holes nobody can explain, or a bill that grew because a loop retried a spend cap for eight hours.

An interviewer might ask: "Your eval harness hit a mix of 429s, 529s and malformed outputs overnight. Walk me through what it should have done with each." A strong answer classifies from the documented error, not the status code alone. It prices retries with 1 − (1 − q)^k, spots the SDK's hidden retries, keeps failed items out of the score while reporting how many there were, and describes how the policy was tested with injected faults. For practice, run the protocol above against your own harness and write up the table.