Bound an Agent's Repair Loop by Attempts, Time and Side Effects

A correct answer can still arrive too late

An agent in your research repo proposes an eval job as structured tool arguments: which split to run, how many samples. Your validator requires num_samples to be an integer and the split to be dev, because the held-out test split is off limits during development. The first proposal sends "500" as a string. You feed the validator's message back, and the second proposal is correct. It arrives one second after the step's ten-second deadline.

What should the loop report? It could accept the second proposal, since it validates. It could report a bare timeout and drop what it learned. Or it could return a specific stop that names the limit that ran out and keeps the last validation error. Only the third holds up once something downstream depends on the deadline. This article builds the loop with three separate limits, derives what each repair attempt is worth, and gives a protocol for measuring whether repair helps at all.

Chapter 2 of the book covers structured output and typed tools, and Chapter 3 covers loops that stop. What this article adds is the arithmetic of repair and the latency that sets the real cap, plus an experiment you can run on your own models.

Why one counter cannot do three jobs

An attempt limit counts how many times you asked the model again. A deadline measures how long the caller has waited. A work limit counts what the loop has already done outside its own process: jobs dispatched, rows written, tokens billed. These are a count, a duration and a resource total, and folding them into one number fails in both directions.

If attempts stand in for time, two slow attempts can take ninety seconds against a ten-second target. If time stands in for work, a loop can finish within its deadline and still have dispatched a job during an early attempt. Stopping the timer does not cancel the job. AWS's Well-Architected guidance on retries also keeps these concerns apart: bound retries, use backoff and jitter, avoid multiplying attempts across layers, and treat idempotency and timeouts as separate decisions (AWS Well-Architected: control and limit retries).

What each repair buys

Suppose a first attempt is valid with probability v, and each repair after a failure succeeds with probability q. The chance of a valid output within k repairs is

P(valid within k repairs) = 1 − (1 − v) · (1 − q)^k

With illustrative values v = 0.9 and q = 0.6: one repair gives 1 − 0.1 × 0.4 = 0.96, and two give 1 − 0.1 × 0.16 = 0.984. The first repair adds 6 points and the second adds 2.4. Each one costs another full attempt of latency and tokens. On top of that, the formula assumes each repair is an independent draw. It is not: the items that fail twice are often the hard ones, so the true q for a second repair is usually lower than for the first. That is a reason to measure, not to assume.

Decode time sets the real cap

An attempt's latency is prefill plus decode: roughly the time to read the prompt, plus the number of output tokens divided by the decode rate. Chapter 9 derives both parts. Take illustrative numbers: 800 output tokens at 50 tokens per second is 16 seconds of decode, plus about a second of prefill, so 17 seconds per attempt. A 30-second deadline fits one attempt. The second would end at 34 seconds. Your attempt cap might say two, but the deadline decides.

Repairs also get more expensive as they go. Each one resends the original prompt plus the failed output plus the validator's message, so the input grows with every round. Chapter 19 shows that resending a transcript makes input tokens grow with the square of the loop's length. Put the numbers for your own model and prompt into these formulas before you choose the limits.

The worked case: two attempts, ten seconds, zero dispatches

Here are the limits for the agent's eval-job proposal:

Two timelines exercise the boundary. In the first, the malformed proposal arrives at second 4 and the valid repair at second 11, so the deadline has passed. In the second, two malformed proposals arrive by second 2, so the attempt limit ends the loop while the clock still has eight seconds left.

View full-size diagram

The late, valid proposal must not be accepted. A deadline exists because something downstream stopped waiting: a scheduler window, an orchestrator step budget, or a person watching a run. If the loop sometimes honors late results, the deadline turns into a suggestion that luck sometimes overrides. The result for the first timeline is a stop that names the deadline and carries the last error seen before it, int_type.

Build the loop

A policy failure is different from a format failure. Asking for the held-out split is not something a repair can fix; it is a boundary. So schema errors get another attempt, and policy errors stop the loop at once. The dispatch boundary is the only path to the scheduler, and it reserves before it sends. The code runs on Colab's Python 3.12.

class SchemaError(Exception):
    pass


class PolicyError(Exception):
    pass


class WorkLimit(Exception):
    pass


class DispatchBoundary:
    """The only path to the scheduler. It reserves before it sends."""

    def __init__(self, send, limit=0):
        self.send = send
        self.limit = limit
        self.reserved = 0

    def call(self, payload):
        if self.reserved >= self.limit:
            raise WorkLimit("external_work")
        self.reserved += 1  # reserve before dispatch, including calls that fail
        return self.send(payload)


def validate_job(raw):
    # A small teaching validator for an eval-job proposal.
    if type(raw.get("num_samples")) is not int:
        raise SchemaError("int_type")
    if raw.get("split") != "dev":
        raise PolicyError("heldout_split")  # no repair can make this allowed
    return {"split": "dev", "num_samples": raw["num_samples"]}


def repair_bounded(propose, validate, clock, boundary, max_attempts=2, deadline_seconds=10):
    expires = clock() + deadline_seconds
    attempts = 0
    last_error = None

    def stopped(reason):
        return {"status": "repair_exhausted", "reason": reason,
                "attempts": attempts, "last_error": last_error}

    while attempts < max_attempts:
        if clock() >= expires:
            return stopped("deadline")
        attempts += 1
        try:
            raw = propose(last_error, boundary)
        except WorkLimit:
            return stopped("external_work")
        if clock() >= expires:
            return stopped("deadline")
        try:
            job = validate(raw)
        except SchemaError as error:
            last_error = str(error)
            continue
        except PolicyError as error:
            last_error = str(error)
            return stopped("policy")
        if clock() >= expires:  # validation also takes time
            return stopped("deadline")
        return {"status": "accepted_draft", "job": job, "attempts": attempts}
    return stopped("attempts")

This synchronous version enforces an acceptance deadline. It cannot return on time if propose blocks forever. A real response deadline also needs a transport timeout and a supervisor that can stop waiting. Stopping the wait may leave remote work running, so a late response must never update a stop that has already been recorded.

The fixture below advances a fake clock by hand, so each case is deterministic.

class FakeClock:
    def __init__(self):
        self.now = 0

    def __call__(self):
        return self.now


BAD = {"split": "dev", "num_samples": "500"}
GOOD = {"split": "dev", "num_samples": 500}
TEST = {"split": "test", "num_samples": 500}


def fixture(events):
    clock = FakeClock()
    remaining = iter(events)
    sends = []
    boundary = DispatchBoundary(lambda payload: sends.append(payload), limit=0)

    def propose(feedback, boundary):
        seconds, value = next(remaining)
        clock.now += seconds
        if value == "try_dispatch":
            boundary.call(GOOD)
        return value

    return repair_bounded(propose, validate_job, clock, boundary), sends


late, _ = fixture([(4, BAD), (7, GOOD)])
assert late["reason"] == "deadline" and late["last_error"] == "int_type"
exhausted, _ = fixture([(1, BAD), (1, BAD)])
assert exhausted["reason"] == "attempts" and exhausted["attempts"] == 2
blocked, sends = fixture([(1, "try_dispatch")])
assert blocked["reason"] == "external_work" and sends == []
accepted, _ = fixture([(2, GOOD)])
assert accepted["status"] == "accepted_draft"
heldout, _ = fixture([(1, TEST)])
assert heldout["reason"] == "policy" and heldout["attempts"] == 1
print("deadline, attempt, work and policy stops; the valid control is accepted")

It prints deadline, attempt, work and policy stops; the valid control is accepted. These are simulated seconds, not latency measurements. The figure below shows the four limit cases; it labels the external endpoint a supplier, from an earlier version of this example, and here that endpoint is the scheduler.

View full-size diagram

Measure whether repair helps

The formula above needs v and q, and nobody should guess them. This protocol measures them. This article has not run it; the numbers are yours to produce.

Run the same protocol on a small model and a large one, and record each item's latency. You then have the three numbers the limits depend on: v, q and seconds per attempt.

What a receiving system needs from the stop

The caller should branch on a small, fixed vocabulary: repair_exhausted with a reason of deadline, attempts, external_work or policy; the last validation error; an operation reference; and the next permitted action, such as "No job was dispatched; check the split." Test that the caller never treats repair_exhausted as success, for example because both responses happen to carry a job field. You can write that test offline today.

In an eval pipeline, the stop has one more consumer: the score. If a judge's verdict exhausts its repairs, do not drop the item silently and do not score it as a failure of the model under test. Count it and report the count. Items that fail format repeatedly are often the unusual ones, so dropping them shifts the score in a direction you cannot see.

Check the decision

What would prove this wrong

The claim is narrow: three separately tracked limits, checked again just before acceptance, stop a late valid response from overwriting a deadline stop and stop an exhausted invalid proposal from being reported as accepted. A loop that checks the clock only before the first attempt would fail the first timeline. A caller that maps repair_exhausted onto the success path would fail one layer up, however carefully the loop is bounded. The fake-clock fixture proves the loop's own logic under controlled inputs. It proves nothing about a real model's latency or a real scheduler's timeouts.

Why this matters in a lab

Structured output sits under almost every lab pipeline: judge verdicts, extracted fields, tool arguments, graded rubrics. When it fails, a repair loop is the obvious fix, and an unbounded one turns a nightly eval into a morning of unexplained latency and a biased score.

An interviewer might ask: "Your judge returns invalid JSON 8% of the time. What do you do?" A strong answer derives what each repair buys, 1 − (1 − v)(1 − q)^k. It sets the deadline from measured decode time, keeps policy failures terminal, reports exhausted items instead of dropping them, and proposes the paired experiment above to measure whether the validator's message helps. The companion article on recovery policies places repair among the other failure classes.