Agent Tool Idempotency: Reconcile an Uncertain Write

The timeout that could mean two different things

Picture an agent placing an order through a supplier's API. The call takes twelve seconds, then the connection drops before a response arrives. If the retry logic calls the same endpoint again with a freshly generated request identifier, the supplier's system may end up recording two orders for one intended purchase, with the warehouse shipping two crates of the same part. The retry implementation must preserve the identity of the original purchase.

A timeout is information about the network, not about the server. When a request hangs and the socket closes, you learn that a response never reached your process. You learn nothing about whether the supplier's system received the request, processed it, and committed a database row before the connection died. The write might have failed. It might have succeeded and the acknowledgment got lost. From the caller's side these two outcomes are indistinguishable, and a program that treats "no response" as "no effect" will eventually duplicate a real order.

This is the problem Malcolm Featonby describes in AWS's Builders Library piece on idempotent APIs: retries are a good way to recover from transient faults, but a retry without a stable request identity can turn a safe recovery mechanism into a source of duplicated side effects (Malcolm Featonby, "Making retries safe with idempotent APIs," AWS Builders' Library). AWS's own fix has a name: an idempotency key, which they call a client request identifier or client token. It is a value that identifies the operation, not the attempt, and every retry of the same operation must carry the same key.

What an idempotency key actually buys you

An idempotency key is a value the caller generates once per logical operation, before the first attempt, and reuses on every retry of that same operation. The receiving system stores that key alongside the result of the first successful processing. When a second request arrives with a key it has already seen, the system does not redo the work. It looks up the stored result and returns it.

This only works if three things hold. First, the key has to be generated at the point where intent is decided, not at the point where a network call happens. If your retry loop mints a new key every time it fires, you have built a very reliable way to create duplicate operations quickly. Second, the receiving system needs a place to durably record "key X has already been handled," and that record needs a uniqueness guarantee strong enough to survive two requests arriving at nearly the same instant. Third, you need a rule for what happens when the same key arrives with different parameters, because that situation means something went wrong upstream and silently proceeding is worse than failing loudly.

None of this replaces sensible retry policy. AWS's Well-Architected guidance on limiting retries is explicit that bounding attempt counts, applying backoff and jitter, and avoiding retry storms across layered services are a separate discipline from idempotency (AWS Well-Architected Framework, REL05-BP03: Control and limit retry calls). You need both. A capped, backed-off retry that reuses the wrong key still duplicates the order; a perfectly keyed request retried without limit still floods a struggling service.

A worked example: an order endpoint backed by SQLite

Here is a small, complete example using only the Python standard library. The database lives in memory and the calls run sequentially in one process. The example tests local row creation and lookup. It sends no supplier request and makes no claim about a distributed service under concurrent load.

import sqlite3
import json

conn = sqlite3.connect(":memory:")
conn.execute("""
    CREATE TABLE orders (
        idempotency_key TEXT PRIMARY KEY,
        payload TEXT NOT NULL,
        status TEXT NOT NULL
    )
""")
conn.commit()

def place_order(key, payload):
    payload_json = json.dumps(payload, sort_keys=True)
    existing = conn.execute(
        "SELECT payload, status FROM orders WHERE idempotency_key = ?",
        (key,),
    ).fetchone()

    if existing is not None:
        stored_payload, status = existing
        if stored_payload == payload_json:
            return {"outcome": "duplicate_ignored", "status": status}
        return {"outcome": "conflict", "status": "rejected_mismatched_payload"}

    conn.execute(
        "INSERT INTO orders (idempotency_key, payload, status) VALUES (?, ?, ?)",
        (key, payload_json, "committed"),
    )
    conn.commit()
    return {"outcome": "created", "status": "committed"}

first = place_order("order-17", {"sku": "widget-a", "qty": 4})
retry_same = place_order("order-17", {"sku": "widget-a", "qty": 4})
retry_changed = place_order("order-17", {"sku": "widget-a", "qty": 9})

print(first)
print(retry_same)
print(retry_changed)

The expected return values are:

The primary key constraint on idempotency_key is doing the real work. It forces the database to reject a second row with the same key before your application logic ever has to reason about a race between two nearly simultaneous inserts. The lookup-before-insert code path is a convenience for returning a friendly response; the constraint is the safety net for the case where two workers hit place_order at the same moment and both pass the lookup before either commits. But this single-threaded script calls place_order sequentially, one call finishing before the next begins, so it never actually creates that race. A real idempotency layer needs a store with a strong uniqueness guarantee under concurrent writers, which this in-memory, one-process SQLite example can illustrate as a design pattern but cannot verify under actual concurrent load.

The conflict case matters as much as the duplicate case. If a retry arrives with qty: 9 under a key that was already committed with qty: 4, something upstream is confused, maybe a stale client, maybe a bug that regenerated the payload before the retry. Silently accepting the new quantity would mean the key no longer identifies one operation; it would just be a label. Rejecting the mismatch surfaces the bug instead of hiding it inside a wrong order.

Agent tool idempotency starts with operation identity

The mechanism above assumes you already have a sensible key. Choosing one is a design decision, not an implementation detail. The key should be generated at the moment the caller commits to an intent, for example when a user clicks "place order," and it should be derived from something that uniquely identifies that intent, such as a cart ID plus a timestamp, or a UUID generated once and stored alongside the pending operation before the first network call is even attempted. If the key is regenerated every time the retry loop fires, you have not built idempotency; you have built a random number generator that happens to sit next to retry logic.

Reconciliation is the other half. When a retry finally gets a response, three things can be true: the operation succeeded on the first attempt and the second call is a safe no-op, the operation failed on the first attempt and the second call is the real first success, or the two attempts disagree and something needs a human or a compensating action. A caller that only checks "did I get a 200" without inspecting which of these three occurred will eventually paper over a conflict it should have escalated.

This connects to a related lesson about state that outlives any single actor. In the course lesson "Memory That Survives the Actor: Handoffs and Permissions", a successor agent that resumes a job cannot assume a claimed external action happened just because a handoff note says it did; it must reconcile that claim against the sending system's own receipt before deciding whether to retry. The handoff there and the retry key here share the same underlying discipline: an uncertain claim about an external effect gets checked against the external system's own record, not assumed from either silence or a hopeful note.

Running the experiment yourself

The following fixture fixes the statement order so that both simulated callers observe an absent key before either writes. It tests that particular interleaving in one connection. It is not a concurrent workload or a test of distributed failure recovery.

import sqlite3


def interleaved_reads(enforce_unique):
    db = sqlite3.connect(":memory:")
    constraint = " PRIMARY KEY" if enforce_unique else ""
    db.execute("CREATE TABLE effects (operation_key TEXT" + constraint + ")")
    observations = [
        db.execute(
            "SELECT operation_key FROM effects WHERE operation_key = ?",
            ("order-17",),
        ).fetchone()
        for _ in range(2)
    ]
    assert observations == [None, None]
    outcomes = []
    for _ in range(2):
        try:
            db.execute("INSERT INTO effects VALUES (?)", ("order-17",))
            db.commit()
            outcomes.append("inserted")
        except sqlite3.IntegrityError:
            db.rollback()
            outcomes.append("rejected")
    count = db.execute("SELECT COUNT(*) FROM effects").fetchone()[0]
    db.close()
    return outcomes, count


assert interleaved_reads(False) == (["inserted", "inserted"], 2)
assert interleaved_reads(True) == (["inserted", "rejected"], 1)
print("Controlled interleaving: two rows without uniqueness, one with it")

The constraint prevents the second row. It does not return the original API result for the losing caller: application code must catch the conflict and look up that result. Nor does it make a separate supplier request atomic with this database write. Those are additional parts of a real execution protocol.

The diagram below adds the piece the code table doesn't show on its own: the order of events across two nearly simultaneous callers, and exactly where the constraint intervenes versus where application logic alone would have let a duplicate slip through.

Use this second diagram alongside the earlier decision flowchart: the first shows what a single request should return once a key's history is known, and this one shows why the database-level constraint, not the lookup step, is what decides the outcome when two requests overlap in time. The interleaving fixture gives a reproducible example of the duplicate-row case. A production test must also exercise your actual connection, transaction and retry behavior.

Check the decision

What would falsify this approach

Both examples concern local SQLite behavior. Neither includes a supplier adapter or a durable record surviving a process restart. They would be falsified as a production design if the uniqueness guarantee could not be enforced across the actual deployment topology: multiple application servers writing to a database that does not support a true unique constraint, a cache-based store without atomic compare-and-set, or a message queue that can redeliver a key after its retention window expires and the original result has already been discarded. Each of those is a real engineering question this SQLite example does not answer, and none of them should be assumed solved just because the small version works as described. The AWS Well-Architected guidance on bounding retries is a reminder that the retry policy sitting on top of this key logic is a distinct, equally necessary layer; getting the key right and then retrying without limits or backoff just moves the failure mode from duplicated writes to overwhelmed dependencies.