Controlled Improvement: Unit B Worked Solution

Instructor worked edition · 90 minutes of dedicated work · 2026-09-26

This is the worked edition of Chapter 17, Unit B. It contains:

complete answers;
the instructor explanation;
holdout cases that the student edition does not show.

Use it after a first attempt, or to rehearse the session. It runs on Google Colab (Python 3.12) or any local Python 3.12+ kernel, using only the standard library.

Minutes	Dedicated work	Saved evidence
0–10	Predict the inflation of the colleague's reported score	Written prediction
10–30	The maximum of $k$ draws, and selection as a maximum	Worked outputs and handoff
30–60	Construct `expected_max_normal` and pass the visible cases	Learner code and grade table
60–70	Predict the winner's curse and check it against simulation	Prediction table and negative control
70–85	Changed-constraint task: the Bradley–Terry gradient	Transfer results
85–90	Explain the result and save evidence	Retained submission

Run the self-contained setup

The unit needs only Python's standard library. The collapsed cell supplies:

STANDARD, a standard normal distribution with .pdf and .cdf;
simulate_selection(true_rates, cases, trials, seed): scores every candidate on cases pass/fail cases, keeps the best, and returns the chosen candidate's average measured score, its average true rate, and their difference, the optimism;
bradley_terry(r_i, r_j), the probability that $i$ is preferred to $j$ , and log_likelihood(comparisons, ratings).

Run setup on every fresh kernel. Your saved work lives in practical-work/ch17-b.

Supplied setup: selection simulator and Bradley–Terry

The complete supplied setup cell is in the downloadable notebook. Run it there before attempting the cells below. This web edition omits only that compressed setup payload.

Commit to a prediction before the examples

The colleague's sixteen candidates may all be equally good, passing each case half the time. Each scored on thirty cases. Before you run anything, write down:

the score you expect the best of them to report;
how that inflation would change with three hundred cases.

python

prediction_notes = {
    "prediction": "Write the best candidate's expected score, and how it changes at 300 cases.",
    "reason": "Name the rule behind that prediction.",
    "falsifier": "Name an observation that would prove the explanation wrong.",
    "revision": "After execution, explain what changed in your understanding.",
}

Selection is a maximum

A candidate's measured pass rate on $n$ cases is its true rate $p$ plus noise with standard deviation $\sigma = \sqrt{p(1-p)/n}$ . Keeping the best of $k$ equally good candidates keeps the largest of $k$ noisy draws. So its measured score exceeds its true rate by about

\text{optimism} \approx \sigma \cdot \mathbb{E}\Bigl[\max_{1 \le i \le k} Z_i\Bigr], \qquad Z_i \sim \mathcal{N}(0, 1).

The maximum of $k$ independent standard normals is at most $x$ exactly when all $k$ are, with probability $\Phi(x)^k$ . Differentiating gives its density, $k\,\varphi(x)\,\Phi(x)^{k-1}$ . Its mean is the integral of $x$ times that density.

Predict how the largest of $k$ draws grows before you run the simulation.

python

intro_rng = random.Random(4)
for intro_k in (1, 2, 4, 16):
    intro_draws = [max(intro_rng.gauss(0, 1) for _ in range(intro_k)) for _ in range(20_000)]
    print(intro_k, "draws: average maximum", round(sum(intro_draws) / len(intro_draws), 2))

Choose an explicit starting point

This unit starts from Unit A's work. To use your own, replace None with the path to your practical-work/ch17-a/ch17-unit-a-handoff-v1.json. Leave it as None to start from the supplied reference. Your submission records which you chose.

python

LEARNER_HANDOFF = None

python

import hashlib
import shutil

COURSE_INPUT = COURSE_WORK / "ch17-unit-a-handoff-v1.json"
if LEARNER_HANDOFF is not None:
    learner_input = Path(LEARNER_HANDOFF).expanduser().resolve()
    if not learner_input.is_file():
        raise FileNotFoundError("The selected learner handoff does not exist")
    if learner_input != COURSE_INPUT.resolve():
        shutil.copy2(learner_input, COURSE_INPUT)
    HANDOFF_ORIGIN = "LEARNER_SELECTED"
else:
    reference_source = "def change_skill(db, candidate):\n    ...  # Supplied reference.\n"
    reference_handoff = {
        "schema": 1,
        "status": "VISIBLE_PASSED",
        "implementation": reference_source,
        "implementation_sha256": hashlib.sha256(reference_source.encode()).hexdigest(),
    }
    COURSE_INPUT.write_text(json.dumps(reference_handoff, indent=2, sort_keys=True) + "\n")
    HANDOFF_ORIGIN = "SUPPLIED_REFERENCE"
print("Starting evidence:", HANDOFF_ORIGIN)

python

handoff = json.loads(COURSE_INPUT.read_text(encoding="utf-8"))
source_matches = isinstance(handoff.get("implementation"), str) and hashlib.sha256(
    handoff["implementation"].encode()
).hexdigest() == handoff.get("implementation_sha256")
handoff_status = (
    "VERIFIED" if source_matches and handoff.get("status") == "VISIBLE_PASSED" else "INVALID"
)
print("UNIT_A_HANDOFF", handoff_status)

Unit A activates a candidate only after it passes. This unit asks what "passes" means when the candidate was chosen because it scored highest.

Main practical: construct, connect and challenge

1. Construct `expected_max_normal`

expected_max_normal(k, grid=4000) must return $\mathbb{E}[\max_{i \le k} Z_i]$ by numerical integration of $x\,k\,\varphi(x)\,\Phi(x)^{k-1}$ from $-8$ to $8$ , with the trapezoidal rule on grid equal steps. It must raise ValueError when k is below 1.

The starter below is the large- $k$ approximation $\sqrt{2 \ln k}$ . It is right for $k = 1$ and badly wrong for the small $k$ an evaluation meets. Run the visible cases, then repair it.

python

def expected_max_normal(k, grid=4000):
    """Mean of the largest of k independent standard normal draws."""
    if k < 1:
        raise ValueError("k must be at least 1")
    low, high = -8.0, 8.0
    step = (high - low) / grid
    total = 0.0
    for i in range(grid + 1):
        x = low + i * step
        weight = 0.5 if i in (0, grid) else 1.0
        total += weight * x * k * STANDARD.pdf(x) * STANDARD.cdf(x) ** (k - 1)
    return total * step

Hint 1: the trapezoidal rule

Sum the integrand at every grid point, giving the two end points half weight, and multiply by the step width.

Hint 2: two exact answers to check against

For $k = 2$ the answer is $1/\sqrt{\pi}$ , and for $k = 3$ it is $3/(2\sqrt{\pi})$ . Both come from integrating by parts.

python

import copy


def outcome(call):
    try:
        return round(call(), 4) + 0.0
    except Exception as error:
        return type(error).__name__


VISIBLE_CASES = [
    ("one draw has mean zero", lambda: expected_max_normal(1), 0.0),
    ("two draws: 1 / sqrt(pi)", lambda: expected_max_normal(2), round(1 / math.sqrt(math.pi), 4)),
    (
        "three draws: 3 / (2 sqrt(pi))",
        lambda: expected_max_normal(3),
        round(3 / (2 * math.sqrt(math.pi)), 4),
    ),
    ("zero draws are refused", lambda: expected_max_normal(0), "ValueError"),
]


def grade(cases):
    rows = []
    for label, call, expected in cases:
        observed = outcome(call)
        rows.append(
            {
                "case": label,
                "expected": expected,
                "observed": observed,
                "status": "PASS" if observed == expected else "FAIL",
            }
        )
    return rows


visible_results = grade(VISIBLE_CASES)
VISIBLE_PASSED = all(row["status"] == "PASS" for row in visible_results)
for visible_row in visible_results:
    print(visible_row["status"], visible_row["case"], "->", visible_row["observed"])
print("VISIBLE_CONTRACT", "PASSED" if VISIBLE_PASSED else "NEEDS_WORK")

2. Connect: predict the colleague's inflation, then simulate it

The cell below predicts the optimism of the best of sixteen equally good candidates as $\sigma \cdot \mathbb{E}[\max_{16}]$ , from your function, at 30, 100 and 300 cases. It then simulates the colleague's procedure 4,000 times at each size.

Two requirements:

At 100 and 300 cases, the prediction must be within 0.02 of the simulation. At 30 the scores are coarse (steps of 1/30), and the normal approximation is looser.
The negative control: with one candidate there is no selection. Both the prediction and the simulated optimism must be about zero.

Predict the optimism at 30 cases.

python

connected = None
if VISIBLE_PASSED:
    table = []
    for cases in (30, 100, 300):
        sigma = math.sqrt(0.5 * 0.5 / cases)
        predicted = sigma * expected_max_normal(16)
        simulated = simulate_selection([0.5] * 16, cases, 4000, seed=cases)["optimism"]
        table.append(
            {
                "cases": cases,
                "predicted": round(predicted, 3),
                "simulated": round(simulated, 3),
                "reported_score": round(0.5 + simulated, 3),
            }
        )
    close = all(abs(r["predicted"] - r["simulated"]) < 0.02 for r in table if r["cases"] >= 100)
    single = simulate_selection([0.5], 100, 4000, seed=1)["optimism"]
    control = abs(expected_max_normal(1)) < 1e-9 and abs(single) < 0.01
    connected = {"table": table, "prediction_matches": close, "negative_control_zero": control}
    for row in table:
        print(row)
    print("prediction within 0.02 at 100 and 300 cases:", close)
    print("one candidate, no optimism:", control, round(single, 4))
else:
    print("CONNECTION_NOT_READY — repair expected_max_normal, then run again.")

The colleague's thirty-case winner reports a score of about 0.66 for candidates that are all 0.5. At three hundred cases the inflation is still about five points. More cases shrink the curse as $1/\sqrt{n}$ , and more candidates grow it as $\mathbb{E}[\max_k]$ ; nothing but a fresh holdout removes it.

Exit ticket

Answer three questions:

Why does keeping the best of sixteen equal candidates produce a score above their shared true rate?
Which way does the inflation move with more cases, and with more candidates?
What must the colleague do before calling the winner's score "what we can expect"?

python

exercise_report = {
    "unit": "ch17-b",
    "attempted": 1,
    "completed": int(VISIBLE_PASSED),
    "failed": int(not VISIBLE_PASSED),
    "skipped": 0,
    "connection": "PASSED" if connected else "NOT_READY",
    "handoff": handoff_status,
}
print("EXERCISE_REPORT=" + json.dumps(exercise_report, sort_keys=True))

Changed-constraint construction: the Bradley–Terry gradient

Allow fifteen minutes: three to predict, eight to implement and check, and four for a case of your own.

For open-ended outputs, labs collect preferences: of two answers, which is better? Bradley–Terry gives each item a rating $r_i$ and models $P(i \succ j) = \sigma(r_i - r_j)$ . For (winner, loser) pairs, the log likelihood is $\sum \log \sigma(r_w - r_\ell)$ , and its derivative for item $i$ is

\frac{\partial \log L}{\partial r_i} = \sum_{\text{comparisons with } i} \bigl(\mathbf{1}[i \text{ won}] - P(i \text{ wins})\bigr),

actual wins minus expected wins. A reward model is trained on exactly this signal.

Implement bt_gradient(comparisons, ratings):

comparisons is a list of (winner, loser) index pairs; ratings a list of floats;
return the gradient as a list, one entry per rating;
raise ValueError for an index outside the ratings, or an item compared with itself;
do not change the inputs.

Hint: each comparison touches two items For a pair with winner <code>w</code> and loser <code>l</code>, let <code>p</code> be the probability that <code>w</code> wins. The winner gains <code>1 - p</code>; the loser loses the same amount.

python

def bt_gradient(comparisons, ratings):
    gradient = [0.0] * len(ratings)
    for winner, loser in comparisons:
        if winner == loser or not (0 <= winner < len(ratings) and 0 <= loser < len(ratings)):
            raise ValueError("each comparison needs two different items that have ratings")
        surprise = 1 - bradley_terry(ratings[winner], ratings[loser])
        gradient[winner] += surprise
        gradient[loser] -= surprise
    return gradient

python

TRANSFER_CASES = [
    ("one win between equals", [[(0, 1)], [0.0, 0.0]], [0.5, -0.5]),
    ("one win each way", [[(0, 1), (1, 0)], [0.0, 0.0]], [0.0, 0.0]),
    ("an expected win teaches less", [[(0, 1)], [math.log(3), 0.0]], [0.25, -0.25]),
    ("three items", [[(0, 1), (0, 2), (2, 1)], [0.0, 0.0, 0.0]], [1.0, -1.0, 0.0]),
    ("an item without a rating is refused", [[(0, 3)], [0.0, 0.0]], {"raises": "ValueError"}),
    ("an item against itself is refused", [[(1, 1)], [0.0, 0.0]], {"raises": "ValueError"}),
]


def run_transfer(candidate, cases):
    observations = []
    for label, arguments, expected in cases:
        supplied = copy.deepcopy(arguments)
        try:
            actual = [round(x, 6) + 0.0 for x in candidate(*supplied)]
        except NotImplementedError:
            actual = {"unfinished": True}
        except Exception as error:
            actual = {"raises": type(error).__name__}
        passed = actual == expected and supplied == arguments
        observations.append(
            {"case": label, "expected": expected, "observed": actual, "passed": passed}
        )
        print("PASS" if passed else "NEEDS_WORK", label, "expected", expected, "observed", actual)
    return observations


transfer_observations = run_transfer(bt_gradient, TRANSFER_CASES)
TRANSFER_PASSED = all(row["passed"] for row in transfer_observations)
print("TRANSFER_STATUS", "PASS" if TRANSFER_PASSED else "NEEDS_WORK")

Design a counterexample and retrieve the mechanism

Add a case with an expected value you worked out by hand, and rerun the driver. Then fit ratings: start from zeros, repeatedly add a small multiple of your gradient, and re-center to mean zero. Use preferences sampled from ratings you choose, and check that the fit recovers them. Finally, explain what a preference grader that always favors the longer answer would teach a model trained against it.

Instructor explanation and additional transfer cases

Three misconceptions come up.

"The best score is our best estimate." It is the best estimate of the luckiest candidate. With sixteen equal candidates on thirty cases, the winner reports about 0.66 for a true 0.5. Report the chosen candidate's score on a holdout.

"More cases remove the curse." They shrink it as $1/\sqrt{n}$ ; they do not remove it. More candidates grow it again. A search over thousands of prompts needs a larger holdout, not a larger development set.

"A reward model is the truth." It is a Bradley–Terry fit to finite, noisy preferences, and optimizing against it is selection against a measurement: the winner's curse at the scale of a whole model. That is reward hacking, and it is why labs penalize drifting far from the original model and evaluate on held-out preferences.

The cases below add a longer set of comparisons and a strongly favored winner.

python

INSTRUCTOR_TRANSFER_CASES = [
    ("wins and losses cancel", [[(0, 1), (1, 0), (0, 1), (1, 0)], [0.0, 0.0]], [0.0, 0.0]),
    ("a favorite winning teaches little", [[(0, 1)], [math.log(99), 0.0]], [0.01, -0.01]),
]
instructor_observations = run_transfer(bt_gradient, INSTRUCTOR_TRANSFER_CASES)
assert TRANSFER_PASSED and all(row["passed"] for row in instructor_observations)

python

holdout_rng = random.Random(16)
for holdout_k in (4, 8):
    holdout_draws = [max(holdout_rng.gauss(0, 1) for _ in range(holdout_k)) for _ in range(40_000)]
    assert abs(sum(holdout_draws) / len(holdout_draws) - expected_max_normal(holdout_k)) < 0.02
holdout_pairs = [(0, 1)] * 6 + [(1, 0)] * 2 + [(2, 0)] * 3 + [(0, 2)] * 3
holdout_ratings = [0.2, -0.5, 0.3]
for holdout_i in range(3):
    bumped = list(holdout_ratings)
    bumped[holdout_i] += 1e-6
    numeric = (
        log_likelihood(holdout_pairs, bumped) - log_likelihood(holdout_pairs, holdout_ratings)
    ) / 1e-6
    assert abs(numeric - bt_gradient(holdout_pairs, holdout_ratings)[holdout_i]) < 1e-4
assert (
    connected is not None and connected["prediction_matches"] and connected["negative_control_zero"]
)
print("HOLDOUT_RESULT=" + json.dumps({"status": "PASSED", "unit": "ch17-b"}, sort_keys=True))

Save your evidence and explain the result

Fill in the prediction notes and your explanation before saving. Include:

the exact observed value, and the input that caused it;
your code's invocation point;
one failed hypothesis;
the strongest claim the evidence still cannot support.

These simulations assume equally good candidates and independent cases. Real candidates differ, and when one is much better the curse is smaller; the chapter's measured prompt search is an example.

python

explanation_notes = {
    "causal_trace": "Explain the input, learner invocation and observed result.",
    "failed_hypothesis": "Describe a prediction the evidence changed.",
    "remaining_limit": "Name the guarantee not established by this experiment.",
}
course_submission = {
    "unit": "ch17-b",
    "planned_minutes": 90,
    "starting_evidence": globals().get("HANDOFF_ORIGIN", "INDEPENDENT_UNIT_A"),
    "prediction": prediction_notes,
    "explanation": explanation_notes,
    "core_report": exercise_report,
    "transfer": transfer_observations,
    "explanation_review": "HUMAN_REVIEW_REQUIRED",
}
submission_path = COURSE_WORK / "ch17-b-submission-v1.json"
submission_path.write_text(
    json.dumps(course_submission, indent=2, sort_keys=True), encoding="utf-8"
)
print("Saved evidence:", submission_path)
print(
    "COURSE_REPORT="
    + json.dumps(
        {
            "unit": "ch17-b",
            "transfer_passed": TRANSFER_PASSED,
            "starting_evidence": course_submission["starting_evidence"],
            "edition": "instructor",
        },
        sort_keys=True,
    )
)

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.