Turn One Failed Agent Run Into a Regression Eval Case

The stalled trial nobody can explain yet

You are running a 200-trial evaluation of a coding agent against a model served on your own GPU box. Trial 37 goes wrong. The model answers the agent's fifth step, and on the sixth it stops responding. No error, no clean timeout, only silence where a response should be. The run log says the cause is unknown. Not "probably a timeout." Unknown.

This happened to this site in a milder form. During a live class demonstration, a local model answered a request and then stopped responding, and the lesson that records it, in the ZEO course, keeps the cause as unknown. We use it here as a case study, because the discipline it teaches is the one an eval team needs: record what happened without deciding why.

Say you write a timeout simulation, force a stop, and confirm the agent logs the failure and stops cleanly instead of inventing an answer. It reproduces the symptom. Does that prove trial 37 was a timeout? It does not. A regression case exists to make sure a specific past failure cannot silently return. If you build it around a guessed cause, it passes even when the real problem comes back in a different form. This article shows how to save the failure so the case stays honest, and how many reruns it takes to claim a flaky failure is fixed.

Three ways to explain silence

A model that answers once and then goes quiet can come from at least three mechanisms, each with a different fix.

Every row can produce the same outside view. Staring harder at the silence will not separate them. Anthropic's guidance on agent evaluations separates the transcript, the record of a trial, from the outcome, the final state of the environment. An outcome does not tell you which path through the system produced it.

Observation, simulation and expected recovery

A regression case has three parts that must never collapse into one:

• the observation: what happened, exactly as recorded, with cause_status: unknown if the cause is unknown;
• one simulation per hypothesis: a constructed scenario that injects a known cause;
• the expected recovery: what the agent must do whatever the cause turns out to be, such as log the failure, stop or retry within a budget, and never fabricate a response.

View full-size diagram

The diagram never lets a simulation write back into cause_status. An executed simulation tests the recovery code under an injected condition. It cannot tell you the cause of trial 37.

Here is a small record that keeps the separation explicit. It is an in-memory teaching example, runs on Python 3.12 including Colab, and makes no claim about any production incident store.

from dataclasses import dataclass, field


@dataclass
class RegressionCase:
    symptom: str
    source_event: str
    source_revision: str
    cause_status: str  # "unknown", "hypothesized", or "confirmed"
    simulation_id: str | None = None
    expected_recovery: str = ""
    unresolved_questions: list[str] = field(default_factory=list)


def build_observed_case() -> RegressionCase:
    return RegressionCase(
        symptom="model answered step 5, then stopped responding at step 6",
        source_event="eval-sweep-0412/trial-37",
        source_revision="agent@3f9c2e1, server config v7",
        cause_status="unknown",
        expected_recovery="log failure, stop within retry budget, no fabricated output",
        unresolved_questions=[
            "server deadline, resource exhaustion, or a rejected request hidden by the wrapper?",
            "did the server process crash, hang, or keep serving other requests?",
        ],
    )


def build_timeout_fixture() -> RegressionCase:
    return RegressionCase(
        symptom="model answered step 5, then stopped responding at step 6",
        source_event="fixture-run-1",
        source_revision="timeout-fixture-v1",
        cause_status="hypothesized",  # never "confirmed" from a fixture alone
        simulation_id="timeout-fixture-v1",
        expected_recovery="log timeout, reconcile, stop within retry budget",
        unresolved_questions=["this fixture tests one hypothesis only"],
    )


observed = build_observed_case()
timeout_case = build_timeout_fixture()
print(observed.cause_status, "vs", timeout_case.cause_status)

It prints unknown vs hypothesized. The observed case does not advance past unknown because a fixture with a plausible name exists.

Now make two different injected causes produce the same caller-visible result, while the diagnostic record keeps them apart:

class AuthRejected(Exception):
    pass


def timeout_supplier():
    raise TimeoutError("injected deadline")


def auth_supplier():
    raise AuthRejected("injected expired credential")


def observe_supplier(supplier):
    try:
        return supplier(), "completed"
    except TimeoutError:
        return None, "deadline: reconcile before retry"
    except AuthRejected:
        return None, "auth: stop and repair credentials"


timeout_view, timeout_diagnostic = observe_supplier(timeout_supplier)
auth_view, auth_diagnostic = observe_supplier(auth_supplier)
assert timeout_view is auth_view is None
assert timeout_diagnostic != auth_diagnostic
assert observed.cause_status == "unknown"
print("Same missing result; different injected causes and diagnostic records")

The two fixtures return the same None and different diagnostics. That is the finding: the symptom alone cannot identify the cause, so the case must keep the lower-level record. This code executes exception paths without a network call or a real wait. It does not show that trial 37 went through such a wrapper.

View full-size diagram

How many reruns make "fixed" believable?

Suppose the stall appears in about 10% of trials. You apply a fix and rerun. How many clean reruns do you need before "fixed" is a claim rather than a hope?

If the fix did nothing, each rerun still fails with probability 0.1, so n clean reruns in a row happen with probability 0.9^n. You want that to be at most 5%:

0.9^n ≤ 0.05, so n ≥ ln 0.05 / ln 0.9 = −2.996 / −0.1054 = 28.4.

You need 29 clean reruns. After 20, an unfixed failure would still show nothing 0.9^20 = 12% of the time. The same arithmetic, read the other way, bounds the new failure rate: zero failures in 30 reruns puts the one-sided 95% upper bound at 1 − 0.05^(1/30) = 9.5%, close to the rule of three, 3 / 30 = 10%. Thirty clean runs cannot show a failure rate below about 10%. That is exactly pass^k from the variance article: the chance that all k trials succeed.

Flaky cases poison a suite

Every case you add has some flake rate of its own. If each of N cases fails spuriously with probability q, independently, a fully green run has probability (1 − q)^N:

• q = 0.01, N = 200: 0.99^200 = 0.134;
• q = 0.001, N = 200: 0.999^200 = 0.819.

A 200-case suite at 1% flake per case is red 87% of the time when nothing regressed. People learn to ignore it, and the regression the suite was built for slips through. Record a measured flake rate per case, rerun failures a fixed number of times before calling them real, and quarantine cases above a flake budget until they are fixed.

Steps for turning a failure into a case a stranger can use

Handing the case to someone who was not there

Someone other than the person who saw the failure will act on it later. A useful handover record carries owner, source_event, cause_status, simulation_ids, rerun_rule and next_decision. For trial 37: the owner is the engineer who runs the eval harness; the source event names the sweep and trial; the cause stays unknown; two fixtures are listed; the rerun rule says 29 clean reruns before closing; the next decision says "compare the server's own logs for trial 37 against both fixtures before assigning a cause." A handover that says "we think it was a timeout, please add a retry" smuggles a hypothesis in as a conclusion.

Check the decision

What would prove this wrong

Lower-level evidence can distinguish the hypotheses. If the server's logs show an out-of-memory kill at the moment of trial 37, that is direct evidence, and cause_status can move to confirmed with the log cited. That does not contradict this article; it is the richer observation the case was waiting for. Check whether your stack records the fingerprints that would separate a deadline, an exhausted GPU and a rejected request. If it does not, the first fix is to the logging, before any fix to the agent.

Why this matters in a lab

Long evaluation sweeps and training runs fail in ways that are rare, expensive to reproduce and easy to misattribute. Teams that turn each failure into a regression case, with the observation kept apart from the hypotheses, a stated rerun rule and a measured flake rate, build suites people trust. An interviewer asking "tell me about a bug you could not reproduce" wants to hear this: what you recorded, which hypotheses you tested separately, how many reruns you needed before calling it fixed, and why. Chapter 17 adds the other half: a candidate change that causes a regression must leave the active version intact, and the evaluation that says so must be preserved.