The approval that was right when it was given
A coding agent in your research repo fixes a bug in how the eval harness seeds its sampler. It opens a branch, and you review the diff against main at commit 3f9c2e1. It looks right, so you approve the merge. Twenty minutes later, when the agent gets to it, a teammate has merged a refactor of the same harness. The agent's diff still applies cleanly. It now changes code you never saw.
Nothing was wrong with your review or with the agent's diff. The problem is the gap between them. Security engineers have a name for this: a time-of-check-to-time-of-use race. The MITRE weakness catalog defines it as a product that "checks the state of a resource before using that resource, but the resource's state can change between the check and the use in a way that invalidates the results of the check" (CWE-367). A human approval is a check. The agent's action is the use.
Chapter 11 of the book builds approval records bound to an exact proposal and rechecks authority at the send boundary. This article adds three things: how to estimate how often an approval goes stale, the tradeoff between comparing IDs and comparing relevant fields, and the atomic primitives that close the last gap in tools you already use.
What an approval should bind
A yes/no flag is too thin. Bind each approval to four things:
- The state it was checked against: the commit of
main, a dataset version, or a snapshot ID. - The exact proposal: a hash of the canonical diff, config or command.
- The reviewer, from an authenticated path. A hash does not prove who approved.
- The action authorized: merge, delete, promote, submit.
The hash and the state ID catch different failures. The hash catches substitution: the agent regenerated its branch after approval and the head moved. The state ID catches staleness: the proposal is the same, but the world around it moved. Neither check does the other's job.
How often does an approval go stale?
Model the relevant changes, such as merges to main, as arriving at random at an average rate λ. The number of changes in a window of length w is then Poisson, and the chance of none is
P(still current after w) = e^(−λ·w)With λ = 3 merges per hour and a review that takes w = 20 minutes, λw = 1, so P = e⁻¹ ≈ 0.37. Nearly two approvals in three would be stale by the time the agent acts, if staleness means "main moved at all." These numbers are illustrative; read λ from your repo's merge history and w from your review queue.
Now suppose only one merge in ten touches the files in the agent's diff. The relevant rate drops to 0.3 per hour, λw = 0.1, and P = e^(−0.1) ≈ 0.905. That is the design tradeoff this article returns to: a gate that compares whole-state IDs is simple and strict, and a gate that compares only relevant fields keeps most approvals valid but needs a correct definition of "relevant."
The same formula sizes the last gap, between the final check and the action itself. If that gap is g = 0.2 seconds and λ = 3 per hour, the chance of a change inside it is about λg = (3/3,600) × 0.2 ≈ 1.7 × 10⁻⁴. That sounds negligible until you multiply by volume: across 10,000 agent actions, you expect about 1.7 to act on stale state. A small gap is not a closed gap.
Why pausing the workflow does not pause the world
Many agent frameworks implement approval as an interrupt: the workflow pauses, a person responds, the workflow resumes. LangGraph's documentation on interrupts warns that resuming an interrupted node can rerun code that ran before the interrupt, so side effects there must be safe to repeat (LangGraph, "interrupts"). The workflow's state is frozen during the pause. The repository, the results table and the cluster are not. Whatever the workflow captured before the pause is a claim about the past.
The recheck, in code
The function below runs immediately before the action. It uses only Python's standard library and runs on Colab's Python 3.12.
import hashlib
import json
def proposal_hash(proposal):
encoded = json.dumps(proposal, sort_keys=True, separators=(",", ":")).encode()
return hashlib.sha256(encoded).hexdigest()
def recheck(approval, main_commit, proposal, action, reviewers):
if approval["reviewer"] not in reviewers or approval["action"] != action:
return "approval_not_authorized"
if approval["main_commit"] != main_commit:
return "stale_approval"
if approval["proposal_hash"] != proposal_hash(proposal):
return "approval_mismatch"
return "eligible"
proposal = {"branch": "agent/fix-eval-seed", "head": "a41c9e0"}
approval = {"main_commit": "3f9c2e1", "proposal_hash": proposal_hash(proposal),
"reviewer": "sam", "action": "merge"}
print(recheck(approval, "3f9c2e1", proposal, "merge", {"sam"}))
print(recheck(approval, "8d04a7b", proposal, "merge", {"sam"}))
print(recheck(approval, "3f9c2e1", {**proposal, "head": "c07d2b8"}, "merge", {"sam"}))
print(recheck(approval, "3f9c2e1", proposal, "delete_branch", {"sam"}))
print(recheck(approval, "3f9c2e1", proposal, "merge", set()))It prints eligible, stale_approval, approval_mismatch, approval_not_authorized and approval_not_authorized. The first line is the positive control. The others each refuse for a different, named reason: main moved, the agent's branch moved, the action is not the one approved, and the reviewer is no longer on the list.
eligible is a decision, not an action. It merges nothing and consumes nothing. Which brings us to the gap.
Close the last gap with compare-and-swap
If you read main, compare it, and then merge in a later line of code, a merge by someone else can land between the two lines. Your check was accurate when you made it and stale when you acted on it. The fix is to make the check part of the write: the write succeeds only if the state is still what you checked.
For local state, a conditional update does this in one statement. The approval is consumed only if main is still at the approved commit and nobody has consumed it yet:
import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE approvals (id TEXT PRIMARY KEY, main_commit TEXT, consumed INTEGER)")
db.execute("INSERT INTO approvals VALUES ('ap-1', '3f9c2e1', 0)")
def consume(db, approval_id, current_main):
with db:
changed = db.execute(
"UPDATE approvals SET consumed = 1 "
"WHERE id = ? AND main_commit = ? AND consumed = 0",
(approval_id, current_main),
).rowcount
return "reserved" if changed == 1 else "refused"
print(consume(db, "ap-1", "8d04a7b"))
print(consume(db, "ap-1", "3f9c2e1"))
print(consume(db, "ap-1", "3f9c2e1"))It prints refused, reserved, refused: refused on a moved main, reserved once on the approved commit, and refused on a second attempt to spend the same approval.
The remote side needs the same primitive, and git has one. The git documentation says --force-with-lease=<refname>:<expect> "will protect <refname> (alone), if it is going to be updated, by requiring its current value to be the same as the specified value <expect>" (git-push). So an agent that updates a shared ref, such as a baseline branch that points at the checkpoint everyone evaluates against, can pass the commit the reviewer approved as the expected value. If anyone moved the ref since, the push is refused at the server. The check and the use happen in one place.
Where the remote system offers no such primitive, you cannot close the gap from your side. Reserve locally, send with a stable operation identity, and reconcile the outcome, as Chapter 12 does.
A test that inspects a stale flag after the write has already happened tests logging, not prevention. Force the race to happen before the write, and assert the write count stayed at zero.
At approval time, store the state ID, the proposal hash, the reviewer and the action together.
Read the current state and recompute the proposal hash independently of the stored approval, and compare all four fields.
Consume the approval with a conditional write, and update remote state with a primitive that takes an expected value. Where none exists, reserve, send with an operation identity, and reconcile.
The figure shows that sequence end to end. It names the remote system a supplier, from an earlier version of this example; here it is the git server or the scheduler.
IDs or relevant fields?
The gate above compares whole-state IDs. It refuses whenever main moved, even if the change was to a README. With three merges an hour and 20-minute reviews, that refuses about 63% of approvals, and people start approving without reading because they are asked so often. That failure is also a safety failure.
A relevant-fields gate compares only what the approval depends on: for a diff, the content of the files it touches, and ideally of what those files import. It keeps about 90% of approvals valid in the example above. The cost is that "relevant" must be defined correctly and tested. A refactor of a helper that the changed file calls is relevant even though the diff never touches it. Start strict, measure your stale-refusal rate, and loosen only with a test for each field you stop comparing.
Check the decision
What would prove this wrong
If approval and action always happen inside one atomic operation, there is no gap, and this gate is unnecessary. Test for that before you build it. If your store has no conditional write, the comparison in application code does not close the race; you need a lock or serialized writes, which cost more. And nothing here is a measured incident rate: the e^(−λw) numbers depend on the rates you put in, and your merge history and review times give you the real ones.
Why this matters in a lab
Agents in research repos increasingly act on shared state after someone approves them: they merge branches, promote checkpoints to a baseline, update eval configs, delete old artifacts. Every one of those is a check followed by a use, often minutes or hours apart.
An interviewer might ask: "A human approves an agent's change, and the agent applies it an hour later. What can go wrong, and how do you prevent it?" A strong answer names the TOCTOU pattern. It estimates staleness with e^(−λw) from real rates, binds the approval to state, proposal, reviewer and action, and closes the final gap with compare-and-swap, naming the primitive, such as a conditional UPDATE or --force-with-lease with an expected value. It also names the ID-versus-fields tradeoff. The companion article on the action boundary covers what should need approval in the first place.
Build approval records that name the exact proposal, reserve spending with the approval, and recheck authority at the send boundary.
