Agent Regression Dataset: Save the Failure, Not the Fix

The stalled run that nobody can explain yet

Here is a small, ordinary problem. During a live class exercise, an agent's local model answered a request, and then, partway through the next one, it stopped responding. No error message, no clean timeout, just silence where a response should have been. The instructor's job sheet says the cause is unknown. Not "probably a timeout," not "likely a memory issue" — unknown. That single word is the whole difficulty of this lesson, because everyone who looks at a stalled model wants to reach for the nearest plausible explanation and start fixing it.

Say you build a timeout simulation: cap the response time artificially, force a stop, and confirm your agent handles it by logging the failure and stopping cleanly instead of inventing an answer. You run it. It reproduces the same visible symptom: request goes in, nothing comes back, agent logs a failure. Does that prove the original stall was a timeout?

It does not, and the reason is worth spelling out precisely, because this is the exact gap between a debugging exercise and a regression test. A regression test's job is to make sure a specific past failure cannot silently reappear. If you write your test around a guessed cause rather than the observed symptom, you get a test that passes even when the real problem returns wearing a different hat. This article works through how to build what I will call an agent regression dataset: a small, structured record of a failure that preserves what actually happened, keeps any hypothesis about the cause clearly labeled as a hypothesis, and stays useful to whoever eventually tries to fix it.

Three ways to explain silence, and why they are not interchangeable

Consider the stalled model again. A model that answers once and then goes quiet could result from at least three distinct mechanisms, and each one calls for a different fix.

Every row in that table can produce something that looks, from the outside, like "the model responded and then stopped responding." A caller watching only the symptom cannot distinguish these by staring harder at the silence. This is the same separation Anthropic's engineering team draws when they describe agent evaluation as needing tasks, environments, trajectories and outcomes kept distinct, with domain-specific criteria and human calibration rather than one aggregate number (Anthropic, "Demystifying evaluations for agents," https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). Their point transfers directly here: an outcome (the stall) does not tell you which trajectory (which underlying mechanism) produced it, and no single pass/fail check settles that question for you.

LangChain's writeup on building agent environments and tasks makes a related, narrower point: an evaluation task needs an input, an environment, and a way to judge the result, and the environment specification should stay separate from any individual task built on top of it (Vivek Trivedy and Nick Hollon, LangChain, "Building agent environments and tasks," https://www.langchain.com/blog/building-agent-environments-and-tasks). Applied to our stalled model, the "environment" is the real, uncertain production path where the failure actually happened. A timeout fixture is a different environment: a constructed one, with a known and injected cause. Confusing the two is exactly how a team ends up believing a bug is fixed because a synthetic test now passes, while the real, still-unidentified mechanism is untouched.

This is the actual teaching case behind this article. A course lesson on agent failure handling records that during a live demonstration, a local model answered and then stopped responding, with the cause left unknown; the lesson explicitly treats any later timeout simulation as a separate constructed exercise, not a diagnosis of what happened (see the When the Model Stops lesson in the ZEO course, /courses/zeo-itam-autumn-2026). That lesson is where the discipline of separating recovery policy from root cause comes from, and it is worth reading in full if you want the broader failure-handling material this article draws one thread from.

Building the case without inventing the cause

The mechanism worth internalizing is this: a regression case has three parts that must never collapse into one. There is the observation (what actually happened, exactly as recorded, with cause marked unknown if it is unknown). There is the simulation (a constructed scenario you build to test a specific hypothesis about what might have caused it). And there is the expected recovery (what your agent should do regardless of which cause turns out to be true — log the failure, stop or retry within a defined budget, never fabricate a response).

Notice what the diagram does not do: it never lets the simulation write back into the cause_status field. An executed simulation can test the implemented recovery under an injected condition. A record describing that simulation cannot. It cannot tell you the cause actually was X.

Here is a small, self-contained way to keep that separation explicit in code. This is a toy in-memory record, not a claim about any production logging system or database; it exists to show the shape of the fields, not to prove anything about how a real deployment stores incidents.

from dataclasses import dataclass, field
from datetime import datetime, timezone

@dataclass
class RegressionCase:
    symptom: str
    source_event: str
    source_revision: str
    cause_status: str  # "unknown", "hypothesized", or "confirmed"
    simulation_id: str | None = None
    expected_recovery: str = ""
    unresolved_questions: list[str] = field(default_factory=list)

def build_observed_case() -> RegressionCase:
    return RegressionCase(
        symptom="model responded once, then stopped responding",
        source_event="class-07-live-demo",
        source_revision="cd7e50c37a76f47",  # shortened lesson revision id
        cause_status="unknown",
        simulation_id=None,
        expected_recovery="log failure, stop within retry budget, no fabricated output",
        unresolved_questions=[
            "was this a server timeout, resource exhaustion, or auth expiry?",
            "did the process crash or hang?",
        ],
    )

def build_timeout_fixture() -> RegressionCase:
    return RegressionCase(
        symptom="model responded once, then stopped responding",
        source_event="fixture-run-1",
        source_revision="timeout-fixture-v1",
        cause_status="hypothesized",  # never "confirmed" from a fixture alone
        simulation_id="timeout-fixture-v1",
        expected_recovery="log timeout, stop within retry budget, no fabricated output",
        unresolved_questions=["this fixture tests one hypothesis only"],
    )

observed = build_observed_case()
timeout_case = build_timeout_fixture()

print(observed.cause_status, "vs", timeout_case.cause_status)

Run this and you get unknown vs hypothesized. That single line of output is the entire discipline of the lesson made visible: the observed case's cause_status never advances past unknown just because a fixture with a plausible name exists. A counter or dataclass instance living inside one Python process proves only what happened inside that process during that run. It says nothing about what happened on the original class demonstration's server, and it should not be mistaken for evidence about it.

Add a second case description for an authorization-failure hypothesis. These dataclasses store labels; neither one has simulated a failed request yet:

def build_auth_fixture() -> RegressionCase:
    return RegressionCase(
        symptom="model responded once, then stopped responding",
        source_event="fixture-run-2",
        source_revision="auth-fixture-v1",
        cause_status="hypothesized",
        simulation_id="auth-fixture-v1",
        expected_recovery="reject and request fresh credentials; retries alone do not help",
        unresolved_questions=["this fixture tests a different hypothesis than the timeout fixture"],
    )

auth_case = build_auth_fixture()
print(timeout_case.expected_recovery)
print(auth_case.expected_recovery)

The output compares two proposed recovery policies. To test the observation problem, we need an adapter that actually handles two different injected failures. This local fixture deliberately gives both failures the same caller-visible result while retaining different diagnostic records:

class AuthRejected(Exception):
    pass


def timeout_supplier():
    raise TimeoutError("injected deadline")


def auth_supplier():
    raise AuthRejected("injected expired credential")


def observe_supplier(supplier):
    try:
        return supplier(), "completed"
    except TimeoutError:
        return None, "deadline: reconcile before retry"
    except AuthRejected:
        return None, "auth: stop and repair credentials"


timeout_view, timeout_diagnostic = observe_supplier(timeout_supplier)
auth_view, auth_diagnostic = observe_supplier(auth_supplier)
assert timeout_view is auth_view is None
assert timeout_diagnostic != auth_diagnostic
assert observed.cause_status == "unknown"
print("Same missing result; different injected causes and diagnostic records")

This executes exceptions and handler branches, without making a network request or waiting for a real timeout. An HTTP authorization rejection ordinarily provides an error response; the deliberate loss of that detail happens in this mock adapter's caller view. The example demonstrates why that impoverished view is insufficient for diagnosis. It does not establish that the class demonstration used such an adapter.

Steps for turning a failure into a case a stranger can use

Handing the case to someone who wasn't in the room

The point of building this record carefully is that someone other than the person who saw the failure has to act on it later, often without the context of the original moment. A useful handover packet needs fields like owner_role, receiving_team, source_event, cause_status, simulation_id, and next_decision. For our example, that might read: owner_role is the integration lead, receiving_team is the operations group that will eventually own the fix, source_event names the class-seven stall, cause_status stays unknown, simulation_id points at whichever fixture is under discussion, and next_decision says "compare the observed trace against both fixtures before assigning a cause."

That last field matters more than it looks. A handover that jumps straight to "we think it was a timeout, please add a retry" smuggles a hypothesis in as a conclusion. A handover that instead says "here is what we observed, here are two candidate explanations we tested, here is what remains unresolved" gives the receiving team the actual state of knowledge, which is usually more useful than false certainty. If the packet arrives without a named owner, the right move is to treat it as incomplete and route it back rather than let an unowned diagnostic claim travel further.

Check the decision

What would prove this wrong, and what to do next

Additional diagnostic evidence can distinguish hypotheses. That is a richer observation than the top-level symptom alone, so it does not contradict the example. That is worth checking for your own system: look at logs below the point where the caller sees silence, and see whether a timeout, a resource crash and an auth rejection actually do leave different fingerprints somewhere you can access. If they do, your regression dataset can use that lower-level signal as real evidence rather than treating cause_status as permanently unknown. If they don't, or you haven't checked, the discipline in this article still applies: record the observation, label every simulation as a hypothesis test, and keep the fields that say "we don't know yet" honest until something better than a matching symptom closes them.

The practical next action is small. Take one failure you have sitting in a log somewhere, right now, and write these four fields: symptom, source revision, cause_status, unresolved questions. Do not build the fixture yet. Just get the observed case down accurately first, because a well-written observation outlives any number of guesses about its cause, and it is the one part of this whole exercise that a later fix cannot rewrite out of the record.