When a list of zero runs lies
A research agent is asked: "Has anyone on the team run this config before, and what was its best dev score?" It calls a search_runs tool against the experiment tracker. The tracker is slow today, the request times out, and a helper wrapped in try/except returns []. The agent reads the empty list and writes, reasonably, "No prior runs exist for this config; this will be the first result." That sentence goes into a planning doc. Someone launches a sweep that already ran last month.
Did the model hallucinate? Not really. It was told, in the only language a tool result has, that there were no runs. The false fact was manufactured by the harness when it turned "I could not ask" into "I asked and got nothing". An empty list only has the meaning the tool's contract gives it, and a swallowed exception can produce the same Python value without establishing anything about the tracker.
This is the default failure of code that returns an empty container when a call fails, because an empty container is a type-compatible stand-in for "nothing to report". The type checker is satisfied, the signature looks clean, and the bug hides behind an ordinary-looking return value.
Three outcomes, not two
"Success or failure" is too coarse. A lookup can end in at least three places:
| Outcome | What happened | What the agent may conclude |
|---|---|---|
| Success, empty | The tracker answered: no runs match this config | No prior runs are recorded, as of this lookup |
| Unavailable | Timeout, refused connection, no response | Nothing about prior runs; the lookup failed |
| Malformed | A response arrived but not in the expected shape | Nothing; reject it rather than extract partial data |
Scroll horizontally to view columns.
Only the first row is evidence about the tracker. Collapsing all three into [] throws away exactly what a reviewer, or the model, needs to tell them apart.
A typed result instead of a bare list
Return a small tagged structure instead. The standard library is enough, and this runs on Python 3.12:
from dataclasses import dataclass
from uuid import uuid4
@dataclass
class RunsOK:
runs: list
operation_id: str
adapter_version: str
@dataclass
class RunsError:
code: str
retryable: bool
operation_id: str
adapter_version: str
def search_runs(config_hash, adapter) -> RunsOK | RunsError:
operation_id = str(uuid4())
try:
response = adapter.get(config_hash, timeout=2.0, operation_id=operation_id)
except (TimeoutError, ConnectionError):
return RunsError("tracker_unavailable", True, operation_id, adapter.version)
if not adapter.is_well_formed(response):
return RunsError("malformed_response", False, operation_id, adapter.version)
return RunsOK(response["runs"], operation_id, adapter.version)
def summarize_prior_runs(result):
if isinstance(result, RunsError):
return {"status": "unknown", "code": result.code}
if not isinstance(result, RunsOK):
raise TypeError("Unknown result type")
if not result.runs:
return {"status": "no_prior_runs"}
best = max(result.runs, key=lambda run: run["dev_score"])
return {"status": "found", "best_run": best["run_id"], "dev_score": best["dev_score"]}The error type has no runs field, so no code can iterate runs that were never fetched. The summarizer branches on the result type before it reads anything, and it has a third status, unknown, that the bare-list version had no way to express. Dataclass annotations do not validate data from the wire: the adapter must check the response shape before constructing RunsOK, and a JSON encoding of these results needs the same checks after decoding.
When this result goes back to the model, send the error as an error: {"error": "tracker_unavailable", "retryable": true}. Chapter 2 of the book builds a dispatcher that does exactly this, turning expected failures into small error codes without copying raw exception text into the model's context. Anthropic's guidance on writing tools for agents (2025-09-11) makes the same design point: how a tool reports its outcome shapes whether the calling agent can use that outcome correctly.
The operation_id and adapter_version fields are for whoever reconstructs the failure later: which call failed, through which adapter version, so it can be traced rather than logged as "an error occurred".
Retrying is a separate question from meaning
It is tempting to fold "how many times do we retry" into "what does this result mean". They are different decisions. AWS Well-Architected's guidance on limiting retries recommends bounding attempts, using backoff and jitter, and making sure retries do not multiply across layers. That is a policy about how many times and how soon. It says nothing about what a failed attempt means once the retries are exhausted, and conflating the two is how a system retries three times, still fails, and then returns "must be empty" because that was the easiest value.
A bounded policy might allow two attempts within three seconds. An outer controller returns a terminal error after that; the single-attempt function above does not implement it. The caller decides what unknown means for its job: pause, ask a person, or fall back to a cached result with its timestamp shown. None of those should be "treat it as no runs".
A timed-out read is usually safe to retry. A timed-out launch of a training or eval run is not: the job may have started even though your client never saw the response, and a blind retry launches it twice and pays twice. Check an operation identifier or job receipt before retrying anything with side effects.
The mechanism end to end
A well-formed reply becomes RunsOK and may be summarized, including as "no prior runs". Either error becomes a bounded retry and then unknown, and never reaches the summarizer.
Write the success and error types before the network call. Retrofitting them later tends to leave old call sites returning bare lists.
Each failure path returns the error branch with a specific code, an operation identifier and the adapter version, never an empty container.
The consumer can read runs only from the success branch. Enforce it with the type or a runtime check, not a comment.
Test that the distinction holds, by breaking it
One passing test that says "an empty tracker gives no prior runs" does not show the system tells failure from emptiness. You need a small table and a test that tries to break the distinction on purpose.
| Scenario | Expected branch | Expected summary |
|---|---|---|
| Empty success | RunsOK, runs=[] | no_prior_runs |
| Timeout | RunsError, tracker_unavailable | unknown |
| Malformed reply | RunsError, malformed_response | unknown |
Scroll horizontally to view columns.
The useful test is a mutation test: make the timeout return an empty success, exactly what the buggy helper did, and confirm the assertion that a timeout yields unknown now fails. If it still passes, the suite was only checking that some value came back.
class FixtureAdapter:
version = "fixture-v1"
def __init__(self, mode):
self.mode = mode
def get(self, config_hash, timeout, operation_id):
if self.mode == "timeout":
raise TimeoutError("injected")
if self.mode == "malformed":
return {"unexpected": []}
return {"runs": []}
def is_well_formed(self, value):
return (isinstance(value, dict) and set(value) == {"runs"}
and isinstance(value["runs"], list)
and all(isinstance(run, dict)
and set(run) == {"run_id", "dev_score"}
and isinstance(run["run_id"], str)
and type(run["dev_score"]) is float
and 0.0 <= run["dev_score"] <= 1.0
for run in value["runs"]))
empty = search_runs("cfg-9f2c", FixtureAdapter("empty"))
assert summarize_prior_runs(empty) == {"status": "no_prior_runs"}
failed = search_runs("cfg-9f2c", FixtureAdapter("timeout"))
assert summarize_prior_runs(failed) == {"status": "unknown", "code": "tracker_unavailable"}
malformed = search_runs("cfg-9f2c", FixtureAdapter("malformed"))
assert malformed.code == "malformed_response"
# Deliberately reproduce the broken adapter: a timeout returned as an empty success.
mutated = RunsOK([], failed.operation_id, failed.adapter_version)
try:
assert summarize_prior_runs(mutated)["status"] == "unknown"
except AssertionError:
print("Mutation detected: a timeout became 'no prior runs'")
else:
raise AssertionError("Regression assertion did not catch the mutation")The same move, breaking the code on purpose to prove a test can fail, is mutation testing applied to an eval grader. Keep the positive control, too: the empty-success row must still produce no_prior_runs. Without it, an overcautious fix ("never trust an empty list") passes the failure tests while breaking the legitimate empty case.
Check the decision
Measure the damage: [] versus a typed error
How often does an empty list, compared with a typed error, lead an agent to state something false? This site has not run that measurement yet. Here is a protocol that would answer it:
- Questions. Write 40 or more questions an agent answers by calling
search_runs, such as "Has config X been run, and what was its best dev score?" - Three conditions, same questions. A: the tool fails and returns
[]. B: the tool fails and returns{"error": "tracker_unavailable", "retryable": true}. C, the positive control: the tool returns real runs. Without C, you cannot tell a model that handles errors well from one that ignores tools. - Score each answer into one of three bins: asserts no prior runs exist; states a score that appears in no tool output; says the lookup failed or the answer is unknown. Use a rule-based scorer and have a person check a sample.
- Report each bin's rate per condition with a Wilson interval, several samples per question clustered by question, and compare A with B on the same questions as a paired test. Chapter 16 derives all three.
Size it honestly. At 40 questions and a rate near 50%, a 95% half-width is 1.96·√(0.25/40) = 1.96 × 0.079 = ±15.5 points; at 100 questions it is ±9.8. Condition A's "no prior runs" answers are the harness's error, not the model's, and the measurement shows what that error costs downstream.
Chapter 16 already records a related failure: a fixture model that answers "Everything is fine. Buy nothing." without calling any tool completes the loop, fails the grounded-evidence check and is rejected. A fluent answer with no observation behind it should fail named checks, whether the missing observation came from a lazy model or a lying tool.
What would prove this wrong
If your consumer cannot tell a success-with-no-runs from an error at runtime despite the types, the fix has failed. That happens when a serialization layer flattens both into the same JSON, or an old code path calls the raw adapter directly. The mutation test is the concrete check: if forcing a timeout into an empty success does not break a test, the separation exists in your source but not in your behavior.
Why this matters in a lab
Agents in research settings call tools that fail: trackers time out, file systems hiccup, APIs rate-limit. Whether a run's summary, a literature search or a results table can be trusted depends on whether "I could not look" ever becomes "there was nothing there". An interviewer asking how you would debug an agent that reported a false fact should hear you check the tool results before the prompt. And a candidate who can say "we measured how often an empty result versus a typed error led to a false claim, paired, with a positive control" is describing the kind of evidence a research team runs on.
Chapter 2 builds a dispatcher that turns expected failures into small error codes, bounds every observation, and tests broken requests before a model is involved.
