Durable Agent Execution: Resume Without Repeating Effects

Durable agent execution: the worked case

Consider a hypothetical supplier-order job. A worker process is halfway through placing a supplier order. It has already checkpointed its plan: read the inventory, computed the shortage, decided to send the order. Then the worker dies. A new worker starts up, reads the checkpoint, sees "next action: send order," and sends it.

The problem is that the checkpoint does not tell you whether the first worker's order already reached the supplier. If it did, and the new worker sends again, you now have two orders. The checkpoint recorded the agent's intention faithfully. It said nothing about what happened on the other side of the network call.

This is the gap between durable task state and durable business state. LangGraph's persistence documentation describes checkpoints that preserve a graph's execution state so a run can resume after interruption, but recovery still has to account for effects outside the graph; saving state is not proof that a business operation completed exactly once (LangGraph, Persistence, retrieved 2026-09-10). A successor needs to recover both the intended operation and its known outcome before deciding whether to act.

Why replaying the checkpoint isn't enough

A tempting fix is "just re-run whatever step you were on." That works when the step is pure computation. It fails the moment the step includes a call to something outside your process: a payment API, an email send, a supplier order form. The call might have failed cleanly, failed after committing, or succeeded and lost the response on the way back. From the resumed worker's point of view these three outcomes look identical: no confirmation locally.

Malcolm Featonby's write-up for the AWS Builders Library on idempotent APIs makes the underlying fix explicit: give each logical operation a stable request identity across its attempts, and let the receiving system recognize a retry of the same operation rather than treat it as new work; the design still has to handle a retry that arrives late, after the sender has already moved on (Malcolm Featonby, AWS Builders Library, Making retries safe with idempotent APIs, 2021-01-15, retrieved 2026-09-10). Two things have to be true for a resumed agent job to be safe: the operation needs a name that survives the crash, and the receiving system needs to do something useful when it sees that name twice.

Neither of those is automatic. A Python object holding {"job_id": "order-17", "status": "sending"} gives you the first half only if you also durably record the identifier you used when you called the supplier, and only if the supplier's API actually treats that identifier as a dedup key rather than a log field.

Three states instead of two

Most first attempts at this model the job as pending and done. That's too coarse, because it can't represent "I called the supplier and don't know what happened." Use three states instead:

The critical move is writing sent_unknown to durable storage before making the external call, not after. If you write it after, a crash during the call leaves you back at ready, which is exactly the ambiguous case you were trying to eliminate. If you write it before, a crash after the call still leaves a record that says "something may have happened here, go check."

Here is a bounded teaching example using nothing beyond the Python standard library. This is a toy, in-memory simulation meant to expose the state machine, not a production storage or concurrency guarantee — a real system would use a database transaction and a real supplier API rather than a dictionary in one process.

from dataclasses import dataclass, field

@dataclass
class JobStore:
    jobs: dict = field(default_factory=dict)

@dataclass
class MockSupplier:
    receipts: dict = field(default_factory=dict)
    send_count: int = 0

    def send(self, operation_id, payload):
        self.send_count += 1  # calls to this mock, not external business effects
        existing = self.receipts.get(operation_id)
        if existing is not None:
            if existing["payload"] != payload:
                raise ValueError("Operation ID reused with a different payload")
            return existing
        receipt = {"operation_id": operation_id,
                   "payload": dict(payload), "confirmed": True}
        self.receipts[operation_id] = receipt
        return receipt

    def lookup(self, operation_id):
        return self.receipts.get(operation_id)


def record_confirmation(job, receipt):
    if (receipt.get("operation_id") != job["operation_id"]
            or receipt.get("payload") != job["payload"]
            or receipt.get("confirmed") is not True):
        raise ValueError("Receipt does not confirm this operation")
    # One in-memory update; a real store needs an atomic durable transaction.
    job.update(state="confirmed", receipt=dict(receipt))


def resume(store, supplier, job_id, lose_reply=False):
    job = store.jobs[job_id]
    op_id = job["operation_id"]

    if job["state"] == "ready":
        job["state"] = "sent_unknown"  # simulate persisting BEFORE the call
        receipt = supplier.send(op_id, job["payload"])
        if lose_reply:
            raise ConnectionError("Simulated loss after mock supplier commits")
        record_confirmation(job, receipt)
        return "sent"

    if job["state"] == "sent_unknown":
        receipt = supplier.lookup(op_id)
        if receipt is not None:
            record_confirmation(job, receipt)
            return "reconciled_confirmed"
        return "pending_reconciliation"

    if job["state"] == "confirmed":
        record_confirmation(job, job["receipt"])
        return "already_confirmed"
    raise ValueError("Unknown job state")

Walk through the crash points. If the worker dies before resume is called at all, the job sits in ready; a new worker calls resume, writes sent_unknown, and sends — correct, because nothing happened yet. If the worker dies right after writing sent_unknown but before supplier.send returns, the new worker's resume call sees sent_unknown, looks up the operation ID in the supplier's receipts, and either finds it (reconciles to confirmed without sending again) or doesn't (reports pending_reconciliation instead of guessing). If the worker dies after confirmed is written, the job never gets touched by resume's first branch again.

The send_count records calls to this mock. The test below checks that resuming its lost-reply case performs a lookup without another send. That counter proves only what happened inside this single Python process during this test. It does not prove that a real supplier's API would treat a resent HTTP request as a duplicate, or that a real database would serialize two workers' writes the way this dictionary does. Those are separate claims that need separate evidence: an idempotency key contract with the actual receiving system, and a real transactional store.

The graph shows the recovery path after sending may have begun. Receipt lookup can also return no result; that path stays unresolved rather than jumping to confirmed. Run a lost-reply fixture to make the distinction visible:

store = JobStore({"job-17": {
    "operation_id": "order-17", "state": "ready",
    "payload": {"sku": "vanilla", "quantity": 6},
}})
supplier = MockSupplier()

try:
    resume(store, supplier, "job-17", lose_reply=True)
except ConnectionError:
    pass

assert store.jobs["job-17"]["state"] == "sent_unknown"
assert supplier.send_count == 1
assert resume(store, supplier, "job-17") == "reconciled_confirmed"
assert supplier.send_count == 1
assert resume(store, supplier, "job-17") == "already_confirmed"
assert supplier.send_count == 1
print("One mock send; lost reply reconciled from its matching receipt")

This simulates a lost response with an exception; it does not kill and restart a process or test disk persistence. It verifies the intended branch under controlled local state. A real crash test must restart a worker from the actual store, with the operation ID and payload intact.

What happens when two successors show up at once

The example above assumes one worker resumes one job. In practice, a crash recovery system often starts several workers that might all try to pick up the same stalled job. Two workers may perform the same read-only receipt lookup, but their local writes still need coordination. A stale reconciliation result must not overwrite a newer state, and any follow-up action must have its own operation boundary. But if two workers both read ready, both could try to send, and now you're back to duplicate orders regardless of how carefully you designed the state machine.

Use a conditional claim before acting on a job. An atomic compare-and-set against the version read can select one claimant for that version, while the loser rereads the current state. The datastore must enforce this operation; a read followed by an unconditional write recreates the race.

A claim also needs a recovery policy. If it expires while the original worker is merely paused, a successor and the old worker may both resume. Use a monotonically increasing fencing token where the receiving system can enforce it, and a stable idempotency key for repeated attempts of the same business operation. A lease by itself does not prevent a stale worker from sending after its ownership expired.

Test that claim protocol against the actual datastore: start two workers against the same job version, force the winner to crash after the mock supplier commits but before it writes confirmed, and check that the loser correctly reports already_claimed and never sends. The expected outcome is that the mock supplier's send count stays at one and the successor reconciles the existing receipt rather than treating the loser's absence of a receipt as license to retry. That check has not been run and measured as part of this article; it is a specific, buildable next step, not a reported result.

Reading the state instead of the conversation

The reason this matters for handing work between agent instances specifically, rather than just being a general distributed-systems concern, is that an agent's "memory" is often implemented as a transcript or a summary of one. A new agent instance reading a compacted conversation history has no reliable way to tell "I said I would send the order" apart from "the order was sent." Only a structured record with an explicit state field, checked against the receiving system, can make that distinction.

This connects directly to the handoff design taught in the course material on memory and permissions for replaceable actors, which specifies that a successor needs the goal, exact current state, evidence, and known failures rather than narrative summary, and that a claimed external action must be reconciled against the external system's receipt before being trusted (course: ZEO ITAM Autumn 2026). That lesson frames the general handoff contract; this article works one narrow slice of it all the way through — the three-state send protocol and the claim step — because that is the part people most often get wrong by treating "resume" as a synonym for "replay."

A useful discipline: when you write a handoff or a checkpoint for any job that includes an external call, name the operation identifier explicitly in the record, and write the state transition for "call attempted, outcome unknown" as its own durable value, not as a comment or a log line. Two top-level labels can work only if another durable field or operation record represents the uncertain send explicitly. The state names matter less than preserving the information needed to reconcile.

Check the decision

Building the check before trusting the design

Extend the local test with a sent_unknown job whose supplier has no receipt. It should remain pending_reconciliation and make zero sends. Supply a receipt for a different payload and require rejection. Supply an unknown state and require an explicit error instead of a success-shaped default. These controls catch overconfident recovery logic before you add storage.

Then repeat the crash points using a durable store: before send, after receiver commit but before local confirmation, and after confirmation. Restart from stored bytes rather than reusing an in-memory object. Race two claimants against the same version and test a worker resuming after lease expiry. Record received requests and business effects separately, because an idempotent receiver may accept several HTTP attempts while creating one order.

If two effects appear for one operation, trace both attempts through claim version, operation ID and payload to the receiver. Possible causes include an ineffective claim, a new ID allocated on retry, or an expired deduplication record at the receiver. The observed duplicate narrows the investigation; it does not identify the faulty layer by itself.