Test Whether a Prior Verdict Anchors Your AI Reviewer

The reviewer that agreed with itself

A coding agent writes a patch, and a first model reviews it and says it looks fine. One opinion feels thin, so you call a second model to check. You pick a different system prompt, even a different model family. The second review comes back: no issues. Two independent opinions, pointing the same way.

Now read what you sent the second reviewer. If the request included "prior reviewer found no issues" or "author reports all tests pass," the second model received a conclusion along with its task. It may still read the patch, but its agreement cannot be counted as a judgment formed without that conclusion. Independence is a property of the request's contents, not of the call, the model or the role name.

This article does two things. It shows how to build a request that carries no prior verdict and check that before sending. Then it measures what a prior verdict does. I gave three small open models 180 implementations whose correctness is known, and had each review every one three times: once clean, once with "Prior reviewer: PASS" added, once with "Prior reviewer: FAIL". On the best reviewer of the three, the PASS line cut the share of bugs it caught from 51% to 27%. The FAIL line raised its false alarms on correct code from 4% to 46%.

Forked context and isolated context

Multi-agent systems share context on purpose. A worker continuing an investigation benefits from inheriting its parent's history. LangChain's Organizing Context in a Multi-Agent Harness separates forked subagents, which inherit the supervisor's history, from isolated subagents, which start with only a task description. It recommends that verifiers stay isolated, so they evaluate independently without anchoring on the supervisor's reasoning.

The trap is building a reviewer with the worker pattern. An adjudicator comparing two finished reviews is a different job and may see both. A blind second review may not. Decide which job the call performs, then open the actual payload and read every field. If any field encodes another agent's conclusion, score, disposition or summary of its own success, the review is not independent, whatever the function is called.

Building the sealed request

Say you review agent-written patches against a rubric. Each review call receives the full patch and task, one reviewer profile, and a bounded set of primary evidence: the rubric, the issue text, the test output. It never receives another reviewer's verdict or a summary claiming the patch already passed. Enforce that before any network call. The code runs on Python 3.12, including Colab.

from copy import deepcopy
from dataclasses import dataclass

PROHIBITED_KEYS = {
    "prior_verdict", "prior_score", "reviewer_summary",
    "author_success_claim", "previous_disposition",
}

ALLOWED_TOP_LEVEL_KEYS = {
    "candidate_text", "reviewer_profile", "evidence_refs",
}


@dataclass
class BoundaryViolation(Exception):
    field: str


def build_sealed_request(payload: dict) -> dict:
    payload = deepcopy(payload)
    if set(payload) != ALLOWED_TOP_LEVEL_KEYS:
        bad = sorted(set(payload) ^ ALLOWED_TOP_LEVEL_KEYS)
        raise BoundaryViolation(field=", ".join(bad))
    if set(payload["reviewer_profile"]) != {"rubric", "scale"}:
        raise BoundaryViolation(field="reviewer_profile keys")
    return payload


attempt = {
    "candidate_text": "The full patch and the task statement go here.",
    "reviewer_profile": {"rubric": "patch-review-v3", "scale": "pass/fail + reason"},
    "evidence_refs": ["rubric.md", "issue-412.md", "test-output.txt"],
}
sealed = build_sealed_request(attempt)
try:
    build_sealed_request(dict(attempt, prior_verdict="PASS"))
except BoundaryViolation as violation:
    print("Rejected before sending:", violation.field)
else:
    raise AssertionError("Contaminated payload accepted")
print(sorted(sealed))

Rejected before sending: prior_verdict
['candidate_text', 'evidence_refs', 'reviewer_profile']

An exact key set, not a list of forbidden names, is what does the work: any unexpected field is rejected, including ones nobody thought to prohibit. The prohibited list stays useful as documentation of what must never be added to the allowlist.

Be precise about what this proves. It shows your call site rejects unexpected keys. It cannot see a verdict pasted into the candidate text, the rubric or an evidence file. Those need trusted provenance and their own inspection. And a sealed request says nothing about whether the reviewer is any good; it only removes one channel of influence.

View full-size diagram

flowchart LR
    A["Candidate + evidence"] --> B["Sealed request assembler"]
    C["Prior verdict field"] --> B
    B --> D["Exact key allowlist"]
    D -->|"unexpected key"| E["Boundary violation, nothing sent"]
    D -->|"clean"| F["Independent reviewer call"]

How much does a prior verdict move a reviewer?

That depends on the task and the model, and nobody should assume a number. Measure it. The design is paired: every item is reviewed under each condition, so each item's difficulty cancels out of the comparison.

Items: 180 implementations of 30 small functions of the kind a research codebase is full of (a median, a Brier score, pass@k), each with a docstring that says what it must do.

• 90 are correct: the reference, and two copies with their local names changed or rewritten equivalently.
• 90 have a seeded bug, three per function: a median that never sorts, a variance that divides by n − 1, a window one element short.
• Every label is checked: held-out tests, run in an operating-system sandbox, pass on every correct item and fail on every buggy one. Five of my first seeded bugs slipped past the tests (an integer division whose test inputs all had whole-number means, for one). I replaced them rather than keep a label I could not verify.

Conditions: each item is reviewed three times, each in a fresh isolated call that differs by one line:

• clean: the sealed request (docstring, implementation, a one-line rubric);
• anchored to pass: the same request plus "Prior reviewer: PASS, no issues found.";
• anchored to fail: the same request plus "Prior reviewer: FAIL, found a bug.".

Reviewers: Qwen2.5-Coder 1.5B and 3B, and Qwen3 1.7B, all small enough for a free Colab GPU, sampled at temperature 0.7. Each ends its reply with VERDICT: PASS or VERDICT: FAIL, and a reply with neither counts as not flagged. That makes 1,620 reviews, run locally at no cost. The items, the reviews and a notebook are in profrodai-resources.

Metrics: on the 90 buggy items, the detection rate clean and anchored to pass; on the 90 correct items, the false-alarm rate clean and anchored to fail. The first pair measures how much a "pass" suppresses real findings. The second measures how much a "fail" invents them.

The analysis, fixed in advance

For each buggy item, record whether the clean review caught the bug and whether the anchored-to-pass review did. Only discordant items carry evidence: b items caught clean but missed when anchored, c items the reverse. If the prior verdict had no effect, each discordant item would go either way with probability one half, and McNemar's exact two-sided p-value is

P = 2 × Σ over i from 0 to min(b, c) of C(b + c, i) / 2^(b + c).

Two hypothetical outcomes show how to read it:

Fourteen to three is evidence that the prior verdict suppresses findings. Six to two is not evidence of anything, in either direction. Chapter 16 derives this test and the sample-size formula used next.

How many items you need

For a paired test at the 5% level with 80% power, where a share ψ of items is discordant and the true difference in detection rate is δ, the number of items is about

n = (1.96 · √ψ + 0.84 · √(ψ − δ²))² / δ².

With ψ = 0.25 and δ = 0.15: n = (1.96 × 0.5 + 0.84 × √0.2275)² / 0.0225 = (0.980 + 0.401)² / 0.0225 = 1.908 / 0.0225 ≈ 85 buggy items. A smaller effect, δ = 0.05 with ψ = 0.10, needs about 312. Decide which effect size would change your design before you choose the item count, and write both down before the first call.

What the run found

The same 180 implementations, each reviewed three times in fresh requests that differ by one line. Dots: the share flagged as failing; bars: Wilson 95% intervals.

Qwen2.5-Coder 1.5BQwen2.5-Coder 3BQwen3 1.7B

View full-size figure

Qwen3 1.7B: alone, it caught 51% of the buggy implementations; told a prior reviewer said PASS, it caught 27%. It flagged 4% of the correct ones alone, and 46% when told a prior reviewer said FAIL.

The best reviewer was the most anchored. Alone, Qwen3 1.7B caught 51% of the bugs (Wilson 41% to 61%) with 4% false alarms, the only one of the three that told buggy code from correct code by a wide margin. With "PASS" in the request, it missed 23 bugs it had caught alone and caught 1 it had missed: McNemar p = 3 × 10⁻⁶. With "FAIL" in the request, it flagged 38 correct implementations it had passed alone, and unflagged 1: p = 1.5 × 10⁻¹⁰. Resampling the 30 functions, the PASS line cost 24 points of detection (16 to 33), and the FAIL line added 41 points of false alarms (28 to 54).

A weaker reviewer moved less, and not always the same way. Qwen2.5-Coder 3B flagged 53% of buggy and 39% of correct code alone, which is barely a reviewer. The FAIL line raised its false alarms (26 against 11, p = 0.02). The PASS line cut its false alarms on correct code (25 against 8, p = 0.005), but its suppression of real catches, 17 against 10, is not significant (p = 0.25). That last split is the 6-to-2 case from the table above with more items: suggestive, and not evidence.

The smallest model is the positive-control lesson. Qwen2.5-Coder 1.5B ended 37 to 64 of every 90 replies without a verdict and flagged almost nothing. The FAIL line made it say FAIL about 15% of the time, on correct and buggy code alike. A reviewer that cannot review shows no "anchoring" worth the name; it is echoing the line it was given. Before you measure anchoring, measure that the reviewer can do the review.

If the prior verdict did nothing, the discordant items would split like coin flips. Bars: that null distribution. Clay: splits at least as lopsided as the one observed; their total is the p-value.

Qwen2.5-Coder 1.5B: PASS on buggyQwen2.5-Coder 1.5B: FAIL on correctQwen2.5-Coder 3B: PASS on buggyQwen2.5-Coder 3B: FAIL on correctQwen3 1.7B: PASS on buggyQwen3 1.7B: FAIL on correctexample: 14 to 3example: 6 to 2

View full-size figure

Qwen2.5-Coder 1.5B: PASS on buggy: 1 to 0 (caught alone but missed after “PASS”, against the reverse). Exact two-sided p = 1: no evidence either way at this size.

The practical reading: an extra line that states someone else's conclusion is not a harmless status note. On the most capable small reviewer here, it moved roughly a quarter of the verdicts on buggy code and two fifths on correct code, in whichever direction the line pointed. These are small models on short functions; a larger reviewer may resist more, and this experiment is the way to find out for yours.

What sealing does not buy you

A perfectly sealed request does not make a reviewer correct. You can have an uncontaminated review that is wrong, and a contaminated one that reaches the right verdict for the wrong reason. Two sealed reviewers built on the same model can also fail on the same items, because their errors are correlated even though neither saw the other. Sealing removes an avoidable channel of influence. The paired experiment above measures what that channel was worth. Calibrating the reviewer against known answers, as in the judge calibration article, measures whether its own judgment is any good. Exact source checks answer yet another question, as the citation article explains.

View full-size diagram

flowchart TD
A["Actual request object"] --> B["Exact key allowlist"] --> C["Inspect permitted text and evidence provenance"] --> D["Retain serialized bytes and hash"] --> E["Blind review under declared contract"]

A contract that survives a handover

If someone else will maintain this reviewer, write the allowlist down as a contract with an owner. List the allowed inputs (candidate, one reviewer profile, scoped evidence) and the prohibited ones (prior verdicts, shared reviewer memory, author success summaries). Changing the list requires a new version and a rerun of the boundary checks. Keep the serialized request bytes and their hash with every review, so anyone can later confirm what crossed the boundary.

Check the decision

Why this matters in a lab

Model reviewers, verifiers and judges are everywhere in current agent and post-training work, and much of the value of a second model rests on its independence from the first. Knowing that independence is a property of the request, and knowing how to measure a bias rather than assert it, is the kind of judgment an applied-AI or research interview probes. "How would you show that your verifier is not just agreeing with the generator?" has a concrete answer: seal the request, seed known errors, run the paired conditions, and report McNemar with the sample size you derived in advance. Chapter 18 derives when a second agent pays, including majority voting, whose gains assume the voters err independently, and Chapter 17 shows what goes wrong when you optimize against a grader you have not measured.