Cancel an Eval Run or Agent Job Without Misreporting What Ran

The dashboard that says "Canceled"

You start an eval of a new checkpoint on a 1,000-item set. Ten minutes in, you notice the system prompt is wrong and press Cancel. The dashboard says "Canceled." Over the next few seconds, dozens of responses still arrive, get billed, and some get written to the results table. Someone opens the table later, sees 500 scored items, and reports the accuracy.

Each handler did what it was written to do. The cancel handler set a flag, the workers stopped taking new items, and the dashboard rendered the flag. The failure is in the model of the job: one bit, "canceled or not," cannot represent "we stopped sending" and "what we already sent finished anyway." For any job that reaches outside its own process, cancellation is a question about timing. The honest answer depends on when each remote call was admitted and when it completed.

This article works through that for eval runs and agent jobs. Chapter 12 of the book derives why a missing reply cannot tell you whether the remote side acted, and Chapter 13 handles the worker that crashes mid-call, with an exercise on canceling while a result is delayed. Here we add three things: how much work is in flight when you cancel, why a partial eval score is biased, and a provider API whose cancellation semantics you can read and copy.

How much is still running when you press cancel

Little's law says that in a stable system, the average number of items in progress equals the arrival rate times the average time each spends inside:

L = λ · W

Suppose your harness sends λ = 20 requests per second and each takes W = 3 seconds end to end. Then L = 20 × 3 = 60 requests are in flight on average. A cancel at a random moment finds about 60 requests that have already left your process. The provider will finish most of them, and you will pay for them.

To price that, use illustrative rates of 3 USD per million input tokens and 15 USD per million output tokens (not a quote for any model). A request with 2,000 input and 500 output tokens costs 2,000 × 3/10⁶ + 500 × 15/10⁶ = 0.006 + 0.0075 = 0.0135 USD. Sixty of them cost about 0.81 USD. That is small for one run. It is not small for an agent job whose in-flight calls launch cluster jobs or write to a shared table, because those effects persist after the cancel.

Two clocks, not one

A cancelable remote action has two event histories. One is internal: when did your process record the cancel request? The other is external: when did the remote system commit its side? You can confirm a stop before dispatch only if cancel and dispatch admission are coordinated, for example under one lock or one conditional database write. Wall-clock timestamps from two machines do not establish that order.

LangGraph's persistence layer is a useful reference because of what it does and does not promise. Checkpoints preserve a graph's execution state so a run can resume after an interruption (LangGraph: persistence). That guarantees your process's internal progress. It does not tell you what a remote system did with a request you already sent. The interrupts documentation adds that resuming an interrupted node can rerun work done before the interrupt, which is why it asks for idempotent side effects (LangGraph: interrupts). A cancel request asks local work to stop. It does not reach into the provider and stop remote work.

A provider API that models this honestly

Anthropic's Message Batches API is a good public example, and it is built for the large evaluations labs run. When you cancel a batch, its processing_status becomes canceling. It ends as ended and may contain partial results for requests processed before the cancellation. Each request then has one of four result types: succeeded, errored, canceled or expired. The documentation says a canceled request is one the batch never sent to the model, and that you are not billed for it.

Look at what the API does not say. It never reports the whole batch as canceled. It reports, per request, whether the work happened. Results can also arrive in any order, so you match them to inputs by custom_id. Your own harness should report the same way.

Name the states instead of guessing

Two states, running and canceled, cannot describe the in-flight case. Track each item separately:

You also need terminal states such as completed and a confirmed failure. A response that arrives after the cancel is still a completed item: it was sent, the model answered, and you were billed. Record it as completed with the cancel request still visible. If you later need to compensate, for example by canceling a cluster job that an agent step launched, that is a new action with its own record.

View full-size diagram

Work the race in code

The simulation below has no network and no real provider. It drives three event orderings directly, so you can see each state transition. It runs on Colab's Python 3.12 with the standard library.

from threading import Lock


class Item:
    """One eval item: one model request, tracked from admission to receipt."""

    def __init__(self, item_id):
        self.item_id = item_id
        self.status = "running"
        self.cancel_requested = False
        self.dispatch_started = False
        self.receipt = None
        self.lock = Lock()


def begin_dispatch(item):
    with item.lock:
        if item.cancel_requested or item.dispatch_started:
            return False
        item.dispatch_started = True
        return True


def request_cancel(item):
    with item.lock:
        item.cancel_requested = True
        if item.receipt is not None:
            return item.status  # keep a confirmed outcome
        item.status = "external_outcome_unknown" if item.dispatch_started else "canceled_before_send"
        return item.status


def record_receipt(item, receipt):
    with item.lock:
        if receipt["item_id"] != item.item_id or receipt["completed"] is not True:
            raise ValueError("receipt does not confirm this item")
        item.receipt = dict(receipt)
        item.status = "completed"  # a cancel request must not discard evidence


queued = Item("item-0417")
assert request_cancel(queued) == "canceled_before_send"
assert begin_dispatch(queued) is False

in_flight = Item("item-0418")
assert begin_dispatch(in_flight) is True
assert request_cancel(in_flight) == "external_outcome_unknown"
# The response arrives after the cancel request, and it was billed.
record_receipt(in_flight, {"item_id": "item-0418", "completed": True, "output_tokens": 412})
assert in_flight.status == "completed" and in_flight.cancel_requested
assert request_cancel(in_flight) == "completed"

lost_reply = Item("item-0419")
assert begin_dispatch(lost_reply)
assert request_cancel(lost_reply) == "external_outcome_unknown"
assert lost_reply.receipt is None
print("queued: stopped; in flight: completed and kept; lost reply: still unknown")

It prints queued: stopped; in flight: completed and kept; lost reply: still unknown. Canceling before admission prevents the send. Canceling after admission leaves the outcome unknown until evidence arrives. A matching response then records completion and keeps the cancel request. The lost reply stays unresolved, which is the honest answer.

The lock protects callers that share this one in-process object. Workers in separate processes need a durable conditional transition, such as an UPDATE ... WHERE status = 'running' that reports how many rows it changed. dispatch_started is set on admission, before the send, on purpose: if the worker crashes between admission and send, the item stays uncertain until reconciliation, rather than being assumed unsent.

Why the partial score is biased

The items that finish before a cancel are not a random sample. Short prompts and short answers finish first, and they are often easier. Suppose, for illustration, the set has 600 short items the model gets right 90% of the time and 400 long items at 60%. The true accuracy is

0.6 × 0.90 + 0.4 × 0.60 = 0.54 + 0.24 = 0.78

If the run is canceled after 500 completions, and 450 of them are short, the partial score is

(450 × 0.90 + 50 × 0.60) / 500 = (405 + 30) / 500 = 0.87

The binomial error bar on 500 items is about 1.96 × √(0.87 × 0.13 / 500) ≈ 0.03. The bias here is 0.09, three times the error bar. Nothing in the error bar warns you, because it measures sampling noise and assumes a random sample. Chapter 16 covers error bars for evals. The rule for canceled runs is simpler: do not report a score from them. Mark the run canceled, keep the completed items for debugging, and rerun the whole set.

These numbers are illustrative, not a measurement. To measure the effect on your own harness, record each item's completion order in a full run, then compute the accuracy of the first k items for several values of k and compare it with the full-set accuracy.

What would prove this design wrong

The design fails if any of these hold:

• The external_outcome_unknown state exists, but the dashboard still shows a flat "Canceled," which collapses the honest state back into one bit.
• Reconciliation treats "no response yet" as "not sent," when a response could be delayed or lost.
• The retry path for external_outcome_unknown resends without an idempotency key, which can create a second remote effect instead of a status check.
• Two workers both believe they own reconciliation for the same item, because nobody assigned it to exactly one worker. Chapter 13 solves this with leases and fenced writes.

View full-size diagram

Write the message a reader needs

The states only help if the words on the dashboard track them exactly:

The last row never says "undone." A teammate reading the log next week should be able to reconstruct what the system knew and when, and how much the canceled run cost, without guessing. The companion article on resuming a job uses the same records to restart work without repeating it.

Check the decision

Why this matters in a lab

Evals, sweeps and agent jobs get canceled all the time: a wrong prompt, a bad checkpoint, a quota about to run out. A research engineer who reports numbers from a canceled run, or who cannot say how much a canceled job cost, loses the team's trust in their tables.

An interviewer might ask: "You cancel a 10,000-item eval halfway. What do you know about cost, and what do you report?" A strong answer uses Little's law to estimate the in-flight work, explains why the partial score is biased and by how much it could be, names the per-item states, and describes how to reconcile each unknown item. For practice, extend the simulation above so a cancel sweeps every item and prints counts per state, then add a lookup function that resolves the unknown ones.