AI Agent Retry Policy: When to Retry, Repair, or Stop

Design an AI agent retry policy around the failure

A request times out. Another is rejected because its credentials expired. A third returns malformed JSON. Putting all three into the same retry loop hides the decision that matters: what must change before another attempt is useful?

A transient read timeout may justify another request after backoff. An explicit authentication rejection needs credential repair or an authorized refresh path. A malformed model response may benefit from bounded validation feedback. Each recovery path still consumes the same operation's time and call budget. Classify the failure first, then decide whether another attempt is permitted and likely to help.

Four failures, four different meanings

Consider four authored failure records an agent might produce. These are teaching fixtures, not observations from a running system, and each is labeled as such throughout.

The reasoning behind each row matters more than the row itself. A write timeout is ambiguous: the client didn't get an answer, but for any action with a side effect, "no answer" is not the same as "no effect." If the action was a purchase or a write, the safe move before retrying is to check whether the operation already happened, using an operation identifier or a receipt, rather than assume failure and repeat the whole action. That's the difference between a retry and an accidental duplicate.

An explicit authentication rejection is a reason to stop resending unchanged credentials. A documented refresh flow may obtain a new token and permit another bounded attempt, but that is a state change, not a blind retry. Distinguish failed authentication from insufficient permission: refreshing a token does not grant a permission the caller never had. If the error cannot be classified from the service contract, retain it and escalate rather than guessing. A schema error sits in the middle. If a model's output doesn't match an expected structure, sometimes a corrective retry that includes the validator's specific complaint produces a fixable response next time. That's different from a blind retry: you're not repeating the request, you're repeating it with new information attached. This deserves a bounded attempt count of its own, separate from the timeout budget, because the failure mode is different even though "try again" is part of the answer for both.

A policy refusal, such as a request that exceeds an approved output budget or asks for something outside scope, is not a fault to recover from at all. It's the system working as designed. Treating it as retryable is the error; the correct move is to surface it as a refusal with the reason attached.

AWS's Well-Architected guidance on limiting retries makes a version of this same point for distributed systems generally: retries should be bounded, use backoff and jitter, and must not be allowed to multiply silently across layers, with idempotency treated as a separate requirement from timeout policy (AWS Well-Architected, "Control and limit retry calls"). That guidance is written for infrastructure calls between services, but the same shape of decision applies to an agent classifying its own tool failures: bound the attempts, don't assume the same policy fits every failure, and don't let the retry mechanism do the job that reconciliation is supposed to do.

The multiplication problem hiding in layered retries

Here is where a design that looks reasonable at each layer becomes unreasonable in aggregate. Say an agent's outer orchestration loop allows three attempts. The SDK permits three total attempts per outer attempt, including its initial call. The transport permits three total attempts per SDK attempt. Each of those choices might have seemed sensible in isolation, written by three different people who never saw each other's code. Multiply them: three times three times three is twenty-seven actual network calls for what looks, from the outside, like a single logical request. That is up to 27 transport sends if failures exhaust every layer. A setting of three retries usually means four total attempts; three such layers could instead yield 64 sends. Read the actual setting semantics. A receiving service with a working idempotency contract can prevent repeated business effects, but that does not remove the extra traffic or waiting time.

This is a proposed operating experiment, not a report of an observed incident: three layers, each independently allowing three attempts, expose a possible amplification to 27 transport calls. Testing for this requires deliberately injecting a transient failure at every layer and counting how many actual calls reach the transport boundary. One approach is to let a single layer own retries and disable retries elsewhere. Another is a shared operation budget checked at every actual send. If an SDK retries below the boundary you instrument, an outer counter will undercount; configure or instrument that SDK before claiming an end-to-end limit.

The classifier chooses a recovery action; it does not itself execute that action or enforce a budget. Trace where its result is consumed. A function returning stop provides no protection if the surrounding exception handler retries anyway.

Building the classifier and testing it honestly

The smallest useful artifact here is a pure function that takes a failure code and returns a recovery action. Nothing about it touches a network, a retry counter, or a real provider. This is a hypothetical, single-process exercise; results below are labeled as expected outputs from a small script, not measurements of any deployed system.

def classify_failure(code):
    routing = {
        "timeout": "reconcile",
        "auth_rejected": "stop",
        "schema_error": "repair",
        "policy_refusal": "refuse",
    }
    return routing.get(code, "unknown_failure")

# Hypothetical test cases, not live model failures
cases = ["timeout", "auth_rejected", "schema_error", "policy_refusal", "rate_limit_exotic"]
for c in cases:
    print(c, "->", classify_failure(c))

Run mentally or on your own machine, the expected output is: timeout -> reconcile, auth_rejected -> stop, schema_error -> repair, policy_refusal -> refuse, and rate_limit_exotic -> unknown_failure. That last line matters as much as the first four. A classifier that defaults unrecognized codes to a retry is quietly assuming every failure it hasn't named yet is safe to loop on. Returning unknown_failure makes the missing classification explicit. The caller must stop or escalate that result; the string alone does not enforce either behavior.

Now test a shared budget at a mock transport boundary. The three loops below reproduce nested retry amplification by exhausting three attempts at every layer. The shared counter is reserved immediately before each mock send, including the initial one.

class BudgetExhausted(Exception):
    pass


def nested_attempts(limit=None):
    sends = 0
    reserved = 0
    try:
        for outer in range(3):
            for sdk in range(3):
                for transport in range(3):
                    if limit is not None and reserved >= limit:
                        raise BudgetExhausted
                    reserved += 1
                    sends += 1
                    # The mock send returns a transient failure every time.
    except BudgetExhausted:
        return sends, "budget_exhausted"
    return sends, "layers_exhausted"


assert nested_attempts() == (27, "layers_exhausted")
assert nested_attempts(3) == (3, "budget_exhausted")
assert nested_attempts(0) == (0, "budget_exhausted")
print("27 unbounded mock sends; 3 with the shared limit")

This fixture counts local mock sends; it has no network or concurrent writers. It tests the amplification arithmetic and placement of the shared guard, not a supplier's observed traffic. A distributed implementation needs atomic reservations so two workers cannot both spend the last available attempt.

The second diagram places the budget check after classification and before transport. Add an operation deadline as a separate limit: three attempts can still take too long if each call waits indefinitely. Backoff and jitter also consume elapsed time.

A useful test sequence for the classifier and its budget wrapper: inject a transient error at every layer of a mocked three-layer stack and assert that the fourth send is refused once the shared budget of three is exhausted. Separately test the caller that consumes the classifier: inject an explicit authentication rejection and require one send followed by a stop. A second unchanged-credential send could indicate misclassification or a retrying wrapper that ignored the classification. Inspect both before assigning the cause.

Writing down what happened, not just what worked

A recovery policy is only as useful as the record it leaves behind. A teaching lesson on agent failure handling makes this point directly: a log containing only successful, accepted results conceals the cost of getting there, and a work record should show attempts, failures, and the final state together rather than overwriting a blocked state with a clean success once one arrives (see the Class 7 materials in the ZEO ITAM Autumn 2026 course). In that same course material, a local model responded during a live demonstration and later stopped responding, with the cause left unknown; the lesson uses that as a prompt to separate recovery policy from root-cause diagnosis, not as a solved incident. Any outage simulation you build to practice this is your own fixture, not a reproduction of that event.

A practical recovery matrix worth keeping alongside the classifier names, for each failure code: the code itself, whether it's retryable, the remaining budget at the time of failure, who owns the escalation if it isn't retryable, and what status the user or downstream system should see. That matrix is what turns "we handle errors" into something an operator can actually audit six months later.

Check the decision

Test the limits of the recovery table

The four categories are a starting point, not a complete error taxonomy. A documented rate-limit response, for example, may provide a retry delay. A transport error may leave the remote operation's status unknown. Keep the original error and operation identifier so a later policy can distinguish them instead of inferring meaning from a generic "failed" label.

A schema-repair path also needs a boundary: send concise validation errors that help correct the proposal, keep trusted policy values outside model control, and count the repair call against the shared budget. A policy refusal must not be converted into instructions for finding a way around the policy. The caller should report the refusal and its stated reason.

For a meaningful integration test, inject failures through the real wrapper stack while replacing the external receiver with a controlled endpoint. Compare the sender's reserved-attempt count with that endpoint's received-request log. If the counts differ, inspect hidden SDK retries, connection failures before arrival and concurrent reservations. Neither count alone explains all of those cases.

The local example establishes the counter behavior for one process. Before rollout, document which layer owns retries, the maximum total sends, the overall deadline and the outcome reported when either limit expires. Those choices make recovery costs and stopping behavior inspectable by the team that will operate the workflow.