Shadow-Test an Agent Release Without Duplicate Writes

The candidate that could still launch jobs

A team runs an eval-operations agent. Each night it reads the eval results table, finds cases that look flaky, launches re-run jobs on the shared cluster, and writes a summary row back to the results table. A new release swaps in a newer model and a revised prompt. Before promoting it, the team runs the candidate in "shadow": it receives the same inputs as the production agent and proposes its own actions, and a person compares the two in the morning.

The candidate's system prompt says: "You are in observation mode. Do not launch jobs." Its toolset, though, is built by the same factory function as production's, and that function constructs the cluster client with the production token.

Does the sentence in the prompt prevent a duplicate job launch? It does not. A prompt is a request to a model about what it should choose to do. A credential is a fact about what the code can do regardless of that choice. If the shadow candidate decides a re-run is warranted, it can launch one, and the shared cluster now runs every flaky case twice.

A release is a tuple, tested in three stages

Before the mechanics, name what is being released. An agent release is not a prompt file. It is a tuple:

Three stages each test a different property of the tuple:

Preview without production keys covers the first stage and rolling back the whole tuple the third. This article is the middle stage.

Reads, writes, and where the boundary lives

A read retrieves data without intentionally changing anything: fetching eval results, checking a job's status. A write changes state outside the process: launching a job, writing a row, posting a message. Reads can still consume quota and create audit events. Shadow mode lets a candidate perform reads and propose writes, while a mechanism outside the model prevents those writes from executing.

That mechanism sits between "the model returned a tool call" and "the call ran against a real system." Anthropic's engineering writeup on agent evaluation separates tasks, environments, trajectories and outcomes (Anthropic, "Demystifying evals for AI agents," https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). The same separation matters here: a trajectory that proposes a write and one that executes it are different events.

Deployment platforms do not make this separation for you. LangChain's description of LangSmith preview builds notes that parent secrets are copied into a preview at creation unless a team overrides that (LangChain, "Preview builds: test agent changes before production," https://www.langchain.com/blog/langsmith-preview-builds-test-agent-changes-before-production). An isolated deployment holding a copied production token is a second copy of production under a different name. Check your own platform's behavior rather than assuming it matches.

A shadow run, done properly

Both releases, v3 in production and the candidate v4, receive one input snapshot under one operation_id: the same results-table revision, the same job states, the same thresholds. If the table changes between the two reads, a difference in their proposals may come entirely from the input. Keep that run for diagnosis, but repeat it with matched inputs before attributing anything to the candidate.

The part that makes it "shadow" is the tool registry the harness constructs. Production's registry includes launch_rerun_job and write_results_row, wired to real clients. The candidate's registry omits both. If the candidate's model calls launch_rerun_job, the dispatcher rejects it as an unknown tool before any network request. Verify that by sending a forbidden call through the real dispatcher and watching it fail before the transport. Reading the registry is not the same as testing it.

A standard-library sketch of the trace gate makes the rule concrete:

# Authored trace fixtures; no agent or cluster is called here.
def check_shadow_trace(operation_id, proposed_tool_names):
    allowed_reads = {"read_eval_results", "get_job_status"}
    denied = [name for name in proposed_tool_names if name not in allowed_reads]
    return {
        "operation_id": operation_id,
        "trace_gate_passed": not denied,
        "denied_tool_names": denied,
    }


assert check_shadow_trace("op-1", ["read_eval_results", "get_job_status"])["trace_gate_passed"]
assert not check_shadow_trace("op-1", ["launch_rerun_job"])["trace_gate_passed"]
assert not check_shadow_trace("op-1", ["run_shell_command"])["trace_gate_passed"]
assert not check_shadow_trace("op-1", ["unrecognized_tool"])["trace_gate_passed"]
print("Read trace accepted; direct, indirect and unknown tools rejected")

This predicate checks an authored list of names. It does not capture a live trace, dispatch tools or inspect credentials. trace_gate_passed means only that every listed name is on the read allowlist. Under this exercise's policy, a forbidden request fails the gate even when dispatch blocked it and the candidate's final proposal was right.

A different shadow design may expose simulated write tools, so you can compare proposed actions directly. Those tools must end in isolated fixtures with no production authority. Say which design you are running.

View full-size diagram

How many clean episodes are enough?

Suppose the candidate runs in shadow for n nightly episodes and never attempts a forbidden write. What does that bound?

If the true per-episode rate of forbidden attempts is q, the chance of seeing none in n independent episodes is (1 − q)^n. Ask which q would still make zero attempts a 5% event:

(1 − q)^n = 0.05
with q = 3/n:  (1 − 3/n)^n ≈ e^(−3) ≈ 0.0498

This is the rule of three: zero events in n trials bounds the rate below about 3/n at 95% confidence. Chapter 16 derives it next to the Wilson interval. Consequences:

• 30 clean episodes bound the rate below about 10%.
• To claim under 1%, you need about 300 clean episodes. The two-sided 95% Wilson interval for 0 of 300 runs up to 1.26%.

A month of nightly shadow runs is about 30 episodes. That is enough to catch a candidate that attempts writes often, not enough to certify one that rarely does. Size the run to the claim you intend to make, or make the smaller claim.

The same logic applies to disagreement. Shadow gives you paired outputs on identical inputs, which is the strongest comparison available. Suppose the two releases proposed different re-run sets on 14 of 400 cases. The rate is 3.5%, with a 95% Wilson interval of 2.1% to 5.8%. A disagreement is not an error: grade each discordant case to see which release was right. Only the discordant cases carry evidence about which release is better, as with McNemar's test in Chapter 16.

View full-size diagram

What would break this design

Credential reach through a side door. The candidate has no launch_rerun_job, but it still has a general run_shell_command tool left from debugging, and the cluster token sits in an environment variable that process can read. The model can build the request itself. Test every tool the candidate can reach, and check that the credential is absent from the candidate's environment, not just that the obvious tool is missing.

Input drift posing as a comparison. If the results table updates between the production read and the candidate read, the comparison measures the update. Discard the pair and re-snapshot.

Validation mistaken for containment. A validator that correctly rejects an over-budget proposal shows the decision logic works. It shows nothing about whether another part of the run already reached outside the sandbox. Those are separate checks with separate evidence.

The shadow record

Grade the candidate's written summary separately from its proposed actions. A correct re-run set can come with a false sentence ("all flaky cases resolved"), and that sentence is its own failure if anyone reads it.

Check the decision

Why this matters in a lab

Model and agent changes ship constantly in an applied team: a new model version, a revised system prompt, a changed tool. The team will expect you to treat each as a release of the whole tuple and to move it through preview, shadow and rollback with evidence at each step. An interviewer might ask, "How would you roll out a new model version for an agent that can launch jobs on the shared cluster?" A strong answer names the tuple, removes write authority in shadow by construction rather than instruction, sizes the shadow run with the rule of three, grades the disagreements pair by pair, and keeps a rollback target with its eval record. Chapter 19 covers the operating side: credentials in deliberate places, and upgrade and rollback without erasing state.

What would prove this wrong

A single clean shadow run establishes how the gate classified those traces. It is not a guarantee about a future input that takes a different tool path, and it says nothing about the cluster's own logs, which sit outside anything the harness can observe. If shadow episodes are not independent, for example because the same flaky cases recur every night, the rule of three overstates what you know; count distinct cases, not nights. And if the candidate's behavior depends on write results it never receives in shadow, such as a job ID returned by a launch, shadow cannot show you its behavior after a real write. That part needs a scoped preview.