Make an Agent's Retried Write Land Exactly Once

A timeout that could mean two things

An agent is scoring an eval and writing each result to a shared results service. The write for item 417 hangs for twelve seconds, then the connection drops. The harness retries, and the service records item 417 twice. Nobody notices until someone asks why the run has 1,003 rows for 1,000 items.

A timeout is information about the network, not the server. You learn that a reply never reached your process. You do not learn whether the server received the write and committed it before the connection died. Chapter 12 of the book proves that no finite exchange of messages removes this uncertainty, and measures what blind retries do. The only fix is on the receiving side: every retry of one intended effect carries the same identity, and the receiver refuses to commit that identity twice.

Malcolm Featonby's piece in the AWS Builders' Library describes the same fix: a client request identifier that names the operation, not the attempt, and stays the same across every retry of that operation (Malcolm Featonby, "Making retries safe with idempotent APIs," AWS Builders' Library). The chapter builds this for a single order. This article carries it into research infrastructure: where the key comes from in an eval or sweep, which writes are safe to repeat without one, how long a key must be, and what happens to your score when you get it wrong.

Name the effect, not the attempt

An idempotency key identifies one intended effect. For an eval result, the effect is "the score for this item, in this run, by this grader." So derive the key from those fields:

key = sha256(run_id | item_id | grader_version)

Three rules follow.

1. Mint the key where intent is decided, from durable data. The run ID, item ID and grader version are in your run manifest before the first call, so every retry, and every resumed worker, computes the same key. Chapter 12 makes the point for agents: a model generates a new tool-call identifier on every attempt, so a key taken from the model's tool call changes on each retry and protects nothing.
2. Include everything that makes it a different effect. If you rescore with a new grader, that is a new effect and needs a new key. If you draw a second sample of the same item at nonzero temperature, that is also a new effect, so add a sample index. Otherwise the second sample collides with the first and looks like a conflict.
3. Decide what a mismatch means. The same key arriving with a different payload means something upstream is confused. Refuse it loudly.

AWS's Well-Architected guidance treats bounded retries with backoff and jitter as a separate discipline from idempotency (AWS Well-Architected, REL05-BP03: Control and limit retry calls). You need both. A well-keyed write retried without limit still floods a struggling service.

Some writes are already safe to repeat

Before you add keys everywhere, check which writes are naturally idempotent. Setting a value is safe to repeat; incrementing it is not.

The append row is the one that bites eval code. Suppose a harness appends one line per item, crashes after 300 items, and on restart begins again from item 1. Now 300 items appear twice. If those 300 were answered correctly 90% of the time and the other 700 at 70%, the true accuracy is

0.3 × 0.9 + 0.7 × 0.7 = 0.27 + 0.49 = 0.76

but the mean over all 1,300 lines is

(600 × 0.9 + 700 × 0.7) / 1,300 = (540 + 490) / 1,300 ≈ 0.792

That is a 3-point error, and at nonzero temperature the duplicated items are two different samples, so you cannot even tell which line to drop. These numbers are illustrative. The fix is to write through a keyed upsert, or to deduplicate by key before computing any score.

A results table that refuses duplicates

The example below uses only Python's standard library and an in-memory SQLite database, and runs on Colab's Python 3.12. It shows a keyed write that returns the stored result on a repeat and refuses a changed payload.

import hashlib
import json
import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("""
    CREATE TABLE results (
        result_key TEXT PRIMARY KEY,
        payload TEXT NOT NULL
    )
""")


def result_key(run_id, item_id, grader):
    raw = f"{run_id}|{item_id}|{grader}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]


def record_result(key, payload):
    payload_json = json.dumps(payload, sort_keys=True)
    with conn:
        existing = conn.execute(
            "SELECT payload FROM results WHERE result_key = ?", (key,)
        ).fetchone()
        if existing is not None:
            if existing[0] == payload_json:
                return "duplicate_ignored"
            return "conflict"
        conn.execute("INSERT INTO results VALUES (?, ?)", (key, payload_json))
        return "created"


key = result_key("run-0930", "item-417", "judge-v3")
print(record_result(key, {"correct": True, "tokens": 412}))
print(record_result(key, {"correct": True, "tokens": 412}))
print(record_result(key, {"correct": False, "tokens": 388}))
print(conn.execute("SELECT COUNT(*) FROM results").fetchone()[0])

It prints created, duplicate_ignored, conflict and 1. The third call is the interesting one. A worker that re-ran item 417 and got a different answer tried to write it under the same key. Accepting it silently would mean the key no longer names one effect. Refusing it surfaces the bug: either the rerun should not have happened, or it is a new sample and needs a sample index in its key.

View full-size diagram

How long must a key be?

Truncating a hash to save space is common, and a short key can collide. For n random keys of b bits, the birthday bound gives the probability of any collision as approximately

P(collision) ≈ n² / 2^(b + 1)

For a million results:

One chance in 560 per million results sounds small, until your lab runs hundreds of millions of results a year across evals and sweeps. A collision here is silent: a new result is refused as a duplicate or flagged as a conflict it is not. Use at least 128 bits for keys that live in a shared store.

Keys also need to live long enough. The receiver must remember a key for longer than any retry of it can arrive, including a retry from a worker that resumes after a crash the next morning. If your store expires keys after an hour and your harness can resume after a day, a late retry is treated as new.

Test the race, not just the repeat

The lookup before the insert gives a friendly answer. It is not what prevents duplicates. When two workers both look up an absent key before either inserts, only the unique constraint stops the second row. The fixture below forces that interleaving in one connection, so the statement order is fixed.

import sqlite3


def interleaved_reads(enforce_unique):
    db = sqlite3.connect(":memory:")
    constraint = " PRIMARY KEY" if enforce_unique else ""
    db.execute("CREATE TABLE effects (result_key TEXT" + constraint + ")")
    observations = [
        db.execute("SELECT result_key FROM effects WHERE result_key = ?", ("k417",)).fetchone()
        for _ in range(2)
    ]
    assert observations == [None, None]
    outcomes = []
    for _ in range(2):
        try:
            db.execute("INSERT INTO effects VALUES (?)", ("k417",))
            db.commit()
            outcomes.append("inserted")
        except sqlite3.IntegrityError:
            db.rollback()
            outcomes.append("rejected")
    count = db.execute("SELECT COUNT(*) FROM effects").fetchone()[0]
    db.close()
    return outcomes, count


assert interleaved_reads(False) == (["inserted", "inserted"], 2)
assert interleaved_reads(True) == (["inserted", "rejected"], 1)
print("two rows without the constraint, one with it")

It prints two rows without the constraint, one with it. The constraint does not hand the losing worker the stored result; your code must catch the conflict and look it up. And it does not make a remote call atomic with this write. If the effect lives in someone else's system, such as a job scheduler, that system must honor the key itself, or you must reconcile as Chapter 12 does.

View full-size diagram

Check the decision

What would prove this wrong

The examples use one SQLite process. They would fail as a production design if the deployment's store cannot enforce uniqueness across writers: several servers writing to a store without a true unique constraint, a cache without atomic compare-and-set, or a queue that redelivers after the key has expired. Each is a real question this small version does not answer.

Why this matters in a lab

Research infrastructure is full of writes that get retried: results tables, experiment trackers, artifact stores, job schedulers. When they double-count, the damage is quiet. A table has a few extra rows, a score moves by a few points, and a comparison between two checkpoints flips.

An interviewer might ask: "Your eval harness crashed at item 6,000 and you resumed it. How do you know the final score counts every item exactly once?" A strong answer names the key and what goes into it, separates naturally idempotent writes from appends and increments, sizes the key with the birthday bound, and describes the crash-and-resume test. The companion article on resuming a job builds the resume side. For practice, the chapter's retry experiment and learner file are in the course repo (profrod_sovereign_agent_ch12_retries_learner.py).