Catch Duplicate Eval Results Before a Dictionary Hides Them

The dictionary that lied by omission

An evaluation run grades 200 cases and appends one line per result to a JSONL file. Case q17 times out on its first attempt and is graded a failure. The harness retries it, the retry passes, and the retry is appended to the same file. The analysis script reads the file and builds a dictionary keyed by case ID, because that is the natural way to look up a result. The dictionary says q17 passed. The pass rate goes up. Nothing in the dictionary records that a failure was there a moment ago.

This essay is in the Food for Thought series because the lesson is bigger than one Python idiom. A lossy step placed before a check destroys the evidence the check needs. It happens with dictionaries, sets, drop_duplicates, joins and group-bys, and in evaluation code it changes the number you report.

The right response depends on what the repeat means. It might be an accidental double write, a retry after a timeout, or an intended second sample to measure variance. Keep the rows long enough to make that decision on purpose. The worked example uses a simple policy: one result per case per run, and repeats are rejected.

Validate the list before building the lookup

A dictionary comprehension like {r.case_id: r.passed for r in results} builds its result key by key, in order. When two rows share a key, the second write overwrites the first. There is no error, no warning, and nothing in the resulting object that records the overwrite. If your validation runs after this conversion, the earlier row is gone. You are checking a fact about the survivor, not about the original list.

This is not a bug in Python. Last write wins is the documented behavior of building a dictionary from key-value pairs. The mistake is architectural: validating an artifact that has already thrown away the information the check needs.

from dataclasses import dataclass
from collections import Counter

@dataclass
class EvalResult:
    case_id: str
    passed: bool

def validate_unique_case_ids(results):
    counts = Counter(r.case_id for r in results)
    duplicates = sorted(case for case, count in counts.items() if count > 1)
    if duplicates:
        raise ValueError(f"duplicate_case_id: {duplicates}")
    return results

def to_dict_unsafe(results):
    # This line is the trap: last write wins, silently.
    return {r.case_id: r.passed for r in results}

rows = [
    EvalResult("q17", False),  # first attempt timed out and was graded a failure
    EvalResult("q18", True),
    EvalResult("q17", True),   # the retry, appended to the same file
]

collapsed = to_dict_unsafe(rows)
print(collapsed)
print("pass rate:", sum(collapsed.values()) / len(collapsed))

try:
    validate_unique_case_ids(rows)
except ValueError as error:
    print(error)

Run on Python 3.12, it prints:

{'q17': True, 'q18': True}
pass rate: 1.0
duplicate_case_id: ['q17']

The collapsed dictionary is indistinguishable from a clean run in which q17 passed on its only attempt. The failed attempt is gone. validate_unique_case_ids, called on the raw list, raises before any collapsing happens and names the offending case.

View full-size diagram

The rejection branch exits before any row disappears. Only a list whose IDs passed the uniqueness rule reaches the dictionary and the statistics.

The pass rate now depends on row order

Reverse the two q17 rows and the dictionary keeps the failure instead. Same file contents, same model, a different reported number, decided by the order in which lines were written. That is why "the dictionary looks right" is not evidence that the input was clean.

View full-size diagram

forward = [EvalResult("q17", False), EvalResult("q17", True)]
reversed_ = [EvalResult("q17", True), EvalResult("q17", False)]
for label, pair in [("forward", forward), ("reversed", reversed_)]:
    print(label, "dictionary keeps", to_dict_unsafe(pair))
    try:
        validate_unique_case_ids(pair)
        print(label, "unexpectedly passed")
    except ValueError as error:
        print(label, "->", error)

forward dictionary keeps {'q17': True}
forward -> duplicate_case_id: ['q17']
reversed dictionary keeps {'q17': False}
reversed -> duplicate_case_id: ['q17']

Both orderings are rejected. Counter records two occurrences of q17 either way, and sorting the duplicate IDs makes the error deterministic. If your implementation accepts one ordering, look for a dictionary conversion or an early return before all rows are checked.

Keeping the duplicates is wrong too

The opposite repair, skipping the dictionary and counting every row, fails differently. Each row adds to n, and the error bar on a pass rate shrinks with n. A 95% interval for a proportion is about 1.96 × √(p(1 − p)/n). With a pass rate of 0.7:

n = 200 real cases:          1.96 × √(0.21 / 200) = 0.0635   (±6.4 points)
same 200 cases, each twice:  1.96 × √(0.21 / 400) = 0.0449   (±4.5 points)

Doubling the rows narrows the interval by a factor of √2 ≈ 1.41, and no new information arrived. Chapter 16 of the book measured exactly this on a real model: fourteen cases each run twice, every case giving the same result both times, so the effective sample size was fourteen, not 28, and the naive standard error was too small by about √2. Repeats are not new cases.

What this does and does not prove

The examples manipulate Python objects in one process. Their outputs show what these functions do with the supplied lists. They cannot tell you whether your harness writes one line per attempt or one per case. Find the actual write and read calls before transferring the result.

The EvalResult dataclass does not validate its annotations at runtime; a caller can construct EvalResult("q17", "yes"). A real input boundary needs type checks before this uniqueness check. Pydantic strict mode can reject a string for a strict boolean field, but two valid rows can still share an ID. Field validation and list-level uniqueness need separate checks.

A boundary table specifies the intended behavior:

The second row points at a stronger check: compare the set of result IDs with the set of case IDs. Equal sets and no repeats means every case has exactly one result. That one comparison catches duplicates, missing results and stray IDs at once.

Deciding what a repeat should mean

Catching a repeat is half the job. The other half is deciding what it means, and that decision belongs to whoever owns the evaluation, not to the validator. Two policies are defensible. Reject: a repeated case in a single-attempt run is an error, and the run is fixed at the source (the harness should give each attempt a stable key, the idea Chapter 12 develops for retries). Or keep every attempt on purpose: if you ran each case several times to measure variance or pass@k, record an attempt number, and report statistics over cases, with the clustered standard error Chapter 16 derives. What is not defensible is letting last-write-wins make the choice by accident.

The companion essay on cleaning a messy dataset with an agent covers the other half of this problem: rows that vanish instead of repeating.

Check the decision

Why this matters in a lab

Evaluation numbers are only as good as the join that produced them. Results files get appended by retries, merged across shards, deduplicated by a well-meaning script, and keyed by ID for analysis. Each of those steps can hide a repeat or a gap, and each moves the reported number or its error bar.

An interviewer for a research-engineering role could ask: your new prompt scores three points higher than last week's. Before you report it, what do you check? A good answer includes that every case has exactly one result in both runs, that the comparison is paired on the same cases, and that the interval was computed over cases rather than rows.