Roll Back an Agent Release as a Whole Tuple, Not One File

The prompt is back. The tool is not.

A team ships a release of its eval-operations agent, the agent that reads nightly eval results and launches re-run jobs for flaky cases. The release changes four things: a newer model, a revised prompt, a different sampling configuration, and a tool schema. launch_rerun_job now requires an eval_set_sha field, so every re-run is pinned to the exact eval-set version that produced the flaky result.

Something goes wrong the next night. Someone reverts the prompt file and calls it a rollback. The old prompt goes out. The tool, unchanged, still requires a field the old prompt never learned to send.

Has the release been rolled back? No. The running system is now a mix nobody tested: old prompt, new tool, new model, new configuration. Depending on the tool's code, the call either fails, or it succeeds with a default that silently re-runs cases against the wrong eval-set version. The second outcome corrupts a results table without raising an error.

A release is a tuple

Treat a release as a tuple, not a file:

If your configuration names the model by an alias rather than a pinned identifier, read your provider's documentation on whether that alias can move. If it can, the tuple can change without a commit, and your rollback target is not what you think it is.

Now count. This release changed four components, each with an old and a new version, so there are 2⁴ = 16 possible tuples. The team evaluated two of them: the old release and the new one. The other fourteen are untested combinations. A rollback that lands on one of them is not a return to known-good. It is a new, unevaluated release shipped under pressure.

The eval record should be keyed by the tuple. A rollback target is known-good only if that exact tuple has an eval record: the model identifier, prompt hash, tool versions and config that were graded together.

A fixture that catches the mixed tuple

Here is a local check for the case above. It tests required-key presence only, and the labels are placeholders, not real hashes:

# Local field-presence fixture. p1, s1 and t1 are labels, not real hashes.
release_v1 = {
    "release_id": "r1",
    "model_id": "model-a",
    "prompt_sha": "p1",
    "schema_sha": "s1",
    "tool_sha": "t1",
    "state_schema_version": 1,
}

release_v2_validator_requires = {"case_ids", "eval_set_sha"}


def check_rollback(prompt_output: dict, validator_required_fields: set) -> bool:
    """Check required-key presence only, not full schema compatibility."""
    return validator_required_fields.issubset(prompt_output.keys())


# The rollback under test: old prompt restored, new validator left in place.
old_prompt_output = {"case_ids": ["c-102", "c-417"]}
print(check_rollback(old_prompt_output, release_v2_validator_requires))  # False

The old prompt's call has one key, case_ids. The live validator requires two. The subset check fails, and check_rollback returns False. The prompt file is byte-for-byte the old one, the tool has not changed since v2, and the combination fails a fixture that costs nothing to write.

Exercise both outcomes, then state the limit of this small check:

assert check_rollback(old_prompt_output, release_v2_validator_requires) is False
assert check_rollback(old_prompt_output, {"case_ids"}) is True
assert check_rollback(
    {"case_ids": ["c-102"], "eval_set_sha": "e18"}, release_v2_validator_requires
) is True
# This deliberately passes the key-only check despite having invalid values.
assert check_rollback({"case_ids": "wrong", "eval_set_sha": None}, release_v2_validator_requires) is True
print("Missing-field mismatch caught; value validation remains a separate gate")

The last assertion keeps the toy from being mistaken for a release test. A real rehearsal runs the actual validator and tool against saved state, including jobs created by the newer release.

The fix is not a better prompt. Roll back to a compatible tuple: p1 with s1 and t1 and the old model identifier, together, or keep s2 behind a tested adapter. Inventing a default eval_set_sha would erase the guarantee the field exists for. If an old proposal cannot be tied to its actual eval-set version, stop and regenerate it.

View full-size diagram

View full-size diagram

Code rollback is not state rollback, and neither is effect rollback

The book's Chapter 19 draws a line that matters here. Replacing code keeps the recorded history: a previous release must understand the current state schema and preserve the authority rules. Restoring an old database so old code can start is a different operation. It can erase local knowledge of work that still exists elsewhere, so it needs reconciliation. Keep the two distinct in the runbook and in the evidence.

LangGraph's persistence layer checkpoints graph execution state so a run can resume where it stopped (LangGraph: persistence). That helps recover an interrupted run. It has no view of what the outside world did in response. If v2 launched 40 re-run jobs on the shared cluster, restoring v1's prompt and schema does not cancel them, and the GPU hours are spent.

So the release manifest travels with a second record: an effects ledger listing each external action by operation ID, its receipt, whether it can be reversed, and who owns any compensating action. A cancellation is a new event in that ledger, not an edit of the original. "We rolled the code back" and "we undid the consequences" are different claims with different evidence.

Rehearse before you need it

The negative control matters as much as the positive one. A rollback check that only tests the happy path has never faced the mistake it exists to catch. check_rollback returning False on the mixed pair is that control doing its job. For the step before rollback in the release sequence, see preview without production keys.

Check the decision

Why this matters in a lab

Applied teams ship model and agent changes as often as code: a new snapshot, a prompt revision, a new tool. When one goes wrong, the pressure is to revert the thing that changed last. A team will expect you to know that the unit of release and rollback is the whole tuple, that "known-good" means "has an eval record", and that effects already caused are handled apart from code. An interviewer might ask, "A model upgrade degraded your agent overnight. Walk me through the rollback." A strong answer names the tuple, rolls back to an evaluated one, checks state compatibility, and reconciles the effects the bad release already caused.

What would prove this wrong

A prompt-only rollback can be valid if the old prompt's outputs satisfy the retained schema, tool behavior and state contract, and that combination has been evaluated. Required-key presence alone does not show that: types, ranges, extra-field rules and state interpretation can still differ. And if two components are truly independent, such as a logging setting and the prompt, the count of risky combinations is smaller than 2ⁿ. Test the combinations you might land on, and write down which ones you have not.