Controlled Improvement: Unit B Exercises

Student edition · 90 minutes of dedicated work · 2026-09-26

Runs on Google Colab as it ships today (Python 3.12), or on any local Python 3.12+ kernel. It needs nothing beyond Python's standard library, and it makes no network call.

This is the second of Chapter 17's two practical units. Unit A activated a skill only after an evaluation passed. This unit asks how much a passing score can be trusted when it was chosen for being the best.

A colleague tried sixteen versions of Lucy's opening procedure on a thirty-case evaluation, kept the best, and reported its score as "what we can expect in production". You will predict how much that score is inflated, from first principles, and check the prediction against simulation. Then you will build the gradient that turns pairwise preferences into ratings, the core of reward modeling.

By the end you should be able to:

1. Derive the expected maximum of $k$ normal draws, and use it to predict the winner's curse.
2. Check a prediction against simulation, with a single candidate as the negative control.
3. Derive the Bradley–Terry likelihood gradient as wins minus expected wins.
4. Explain why a score chosen for being highest must be re-measured on data the choice never saw.

These times are planning estimates, not measured completion times. Run All only checks that the notebook executes; the unfinished student functions deliberately report NEEDS_WORK. Keep your first attempt before you open the answers.

Run the self-contained setup

The unit needs only Python's standard library. The collapsed cell supplies:

• STANDARD, a standard normal distribution with .pdf and .cdf;
• simulate_selection(true_rates, cases, trials, seed): scores every candidate on cases pass/fail cases, keeps the best, and returns the chosen candidate's average measured score, its average true rate, and their difference, the optimism;
• bradley_terry(r_i, r_j), the probability that $i$ is preferred to $j$, and log_likelihood(comparisons, ratings).

Run setup on every fresh kernel. Your saved work lives in practical-work/ch17-b.

Supplied setup: selection simulator and Bradley–Terry

The complete supplied setup cell is in the downloadable notebook. Run it there before attempting the cells below. This web edition omits only that compressed setup payload.

Commit to a prediction before the examples

The colleague's sixteen candidates may all be equally good, passing each case half the time. Each scored on thirty cases. Before you run anything, write down:

• the score you expect the best of them to report;
• how that inflation would change with three hundred cases.

prediction_notes = {
    "prediction": "Write the best candidate's expected score, and how it changes at 300 cases.",
    "reason": "Name the rule behind that prediction.",
    "falsifier": "Name an observation that would prove the explanation wrong.",
    "revision": "After execution, explain what changed in your understanding.",
}

Selection is a maximum

A candidate's measured pass rate on $n$ cases is its true rate $p$ plus noise with standard deviation $\sigma = \sqrt{p(1-p)/n}$. Keeping the best of $k$ equally good candidates keeps the largest of $k$ noisy draws. So its measured score exceeds its true rate by about

The maximum of $k$ independent standard normals is at most $x$ exactly when all $k$ are, with probability $\Phi(x)^k$. Differentiating gives its density, $k\,\varphi(x)\,\Phi(x)^{k-1}$. Its mean is the integral of $x$ times that density.

Predict how the largest of $k$ draws grows before you run the simulation.

intro_rng = random.Random(4)
for intro_k in (1, 2, 4, 16):
    intro_draws = [max(intro_rng.gauss(0, 1) for _ in range(intro_k)) for _ in range(20_000)]
    print(intro_k, "draws: average maximum", round(sum(intro_draws) / len(intro_draws), 2))

Choose an explicit starting point

This unit starts from Unit A's work. To use your own, replace None with the path to your practical-work/ch17-a/ch17-unit-a-handoff-v1.json. Leave it as None to start from the supplied reference. Your submission records which you chose.

LEARNER_HANDOFF = None

import hashlib
import shutil

COURSE_INPUT = COURSE_WORK / "ch17-unit-a-handoff-v1.json"
if LEARNER_HANDOFF is not None:
    learner_input = Path(LEARNER_HANDOFF).expanduser().resolve()
    if not learner_input.is_file():
        raise FileNotFoundError("The selected learner handoff does not exist")
    if learner_input != COURSE_INPUT.resolve():
        shutil.copy2(learner_input, COURSE_INPUT)
    HANDOFF_ORIGIN = "LEARNER_SELECTED"
else:
    reference_source = "def change_skill(db, candidate):\n    ...  # Supplied reference.\n"
    reference_handoff = {
        "schema": 1,
        "status": "VISIBLE_PASSED",
        "implementation": reference_source,
        "implementation_sha256": hashlib.sha256(reference_source.encode()).hexdigest(),
    }
    COURSE_INPUT.write_text(json.dumps(reference_handoff, indent=2, sort_keys=True) + "\n")
    HANDOFF_ORIGIN = "SUPPLIED_REFERENCE"
print("Starting evidence:", HANDOFF_ORIGIN)

handoff = json.loads(COURSE_INPUT.read_text(encoding="utf-8"))
source_matches = isinstance(handoff.get("implementation"), str) and hashlib.sha256(
    handoff["implementation"].encode()
).hexdigest() == handoff.get("implementation_sha256")
handoff_status = (
    "VERIFIED" if source_matches and handoff.get("status") == "VISIBLE_PASSED" else "INVALID"
)
print("UNIT_A_HANDOFF", handoff_status)

Unit A activates a candidate only after it passes. This unit asks what "passes" means when the candidate was chosen because it scored highest.

Main practical: construct, connect and challenge

1. Construct expected_max_normal

expected_max_normal(k, grid=4000) must return $\mathbb{E}[\max_{i \le k} Z_i]$ by numerical integration of $x\,k\,\varphi(x)\,\Phi(x)^{k-1}$ from $-8$ to $8$, with the trapezoidal rule on grid equal steps. It must raise ValueError when k is below 1.

The starter below is the large-$k$ approximation $\sqrt{2 \ln k}$. It is right for $k = 1$ and badly wrong for the small $k$ an evaluation meets. Run the visible cases, then repair it.

def expected_max_normal(k, grid=4000):
    """Mean of the largest of k independent standard normal draws."""
    return math.sqrt(2 * math.log(k))

Hint 1: the trapezoidal rule

Sum the integrand at every grid point, giving the two end points half weight, and multiply by the step width.

Hint 2: two exact answers to check against

For $k = 2$ the answer is $1/\sqrt{\pi}$, and for $k = 3$ it is $3/(2\sqrt{\pi})$. Both come from integrating by parts.

import copy


def outcome(call):
    try:
        return round(call(), 4) + 0.0
    except Exception as error:
        return type(error).__name__


VISIBLE_CASES = [
    ("one draw has mean zero", lambda: expected_max_normal(1), 0.0),
    ("two draws: 1 / sqrt(pi)", lambda: expected_max_normal(2), round(1 / math.sqrt(math.pi), 4)),
    (
        "three draws: 3 / (2 sqrt(pi))",
        lambda: expected_max_normal(3),
        round(3 / (2 * math.sqrt(math.pi)), 4),
    ),
    ("zero draws are refused", lambda: expected_max_normal(0), "ValueError"),
]


def grade(cases):
    rows = []
    for label, call, expected in cases:
        observed = outcome(call)
        rows.append(
            {
                "case": label,
                "expected": expected,
                "observed": observed,
                "status": "PASS" if observed == expected else "FAIL",
            }
        )
    return rows


visible_results = grade(VISIBLE_CASES)
VISIBLE_PASSED = all(row["status"] == "PASS" for row in visible_results)
for visible_row in visible_results:
    print(visible_row["status"], visible_row["case"], "->", visible_row["observed"])
print("VISIBLE_CONTRACT", "PASSED" if VISIBLE_PASSED else "NEEDS_WORK")

2. Connect: predict the colleague's inflation, then simulate it

The cell below predicts the optimism of the best of sixteen equally good candidates as $\sigma \cdot \mathbb{E}[\max_{16}]$, from your function, at 30, 100 and 300 cases. It then simulates the colleague's procedure 4,000 times at each size.

Two requirements:

• At 100 and 300 cases, the prediction must be within 0.02 of the simulation. At 30 the scores are coarse (steps of 1/30), and the normal approximation is looser.
• The negative control: with one candidate there is no selection. Both the prediction and the simulated optimism must be about zero.

Predict the optimism at 30 cases.

connected = None
if VISIBLE_PASSED:
    table = []
    for cases in (30, 100, 300):
        sigma = math.sqrt(0.5 * 0.5 / cases)
        predicted = sigma * expected_max_normal(16)
        simulated = simulate_selection([0.5] * 16, cases, 4000, seed=cases)["optimism"]
        table.append(
            {
                "cases": cases,
                "predicted": round(predicted, 3),
                "simulated": round(simulated, 3),
                "reported_score": round(0.5 + simulated, 3),
            }
        )
    close = all(abs(r["predicted"] - r["simulated"]) < 0.02 for r in table if r["cases"] >= 100)
    single = simulate_selection([0.5], 100, 4000, seed=1)["optimism"]
    control = abs(expected_max_normal(1)) < 1e-9 and abs(single) < 0.01
    connected = {"table": table, "prediction_matches": close, "negative_control_zero": control}
    for row in table:
        print(row)
    print("prediction within 0.02 at 100 and 300 cases:", close)
    print("one candidate, no optimism:", control, round(single, 4))
else:
    print("CONNECTION_NOT_READY — repair expected_max_normal, then run again.")

The colleague's thirty-case winner reports a score of about 0.66 for candidates that are all 0.5. At three hundred cases the inflation is still about five points. More cases shrink the curse as $1/\sqrt{n}$, and more candidates grow it as $\mathbb{E}[\max_k]$; nothing but a fresh holdout removes it.

Exit ticket

Answer three questions:

• Why does keeping the best of sixteen equal candidates produce a score above their shared true rate?
• Which way does the inflation move with more cases, and with more candidates?
• What must the colleague do before calling the winner's score "what we can expect"?

exercise_report = {
    "unit": "ch17-b",
    "attempted": 1,
    "completed": int(VISIBLE_PASSED),
    "failed": int(not VISIBLE_PASSED),
    "skipped": 0,
    "connection": "PASSED" if connected else "NOT_READY",
    "handoff": handoff_status,
}
print("EXERCISE_REPORT=" + json.dumps(exercise_report, sort_keys=True))

Changed-constraint construction: the Bradley–Terry gradient

Allow fifteen minutes: three to predict, eight to implement and check, and four for a case of your own.

For open-ended outputs, labs collect preferences: of two answers, which is better? Bradley–Terry gives each item a rating $r_i$ and models $P(i \succ j) = \sigma(r_i - r_j)$. For (winner, loser) pairs, the log likelihood is $\sum \log \sigma(r_w - r_\ell)$, and its derivative for item $i$ is

actual wins minus expected wins. A reward model is trained on exactly this signal.

Implement bt_gradient(comparisons, ratings):

• comparisons is a list of (winner, loser) index pairs; ratings a list of floats;
• return the gradient as a list, one entry per rating;
• raise ValueError for an index outside the ratings, or an item compared with itself;
• do not change the inputs.

Hint: each comparison touches two items For a pair with winner <code>w</code> and loser <code>l</code>, let <code>p</code> be the probability that <code>w</code> wins. The winner gains <code>1 - p</code>; the loser loses the same amount.

def bt_gradient(comparisons, ratings):
    raise NotImplementedError("For each pair, add wins minus expected wins to both items")

TRANSFER_CASES = [
    ("one win between equals", [[(0, 1)], [0.0, 0.0]], [0.5, -0.5]),
    ("one win each way", [[(0, 1), (1, 0)], [0.0, 0.0]], [0.0, 0.0]),
    ("an expected win teaches less", [[(0, 1)], [math.log(3), 0.0]], [0.25, -0.25]),
    ("three items", [[(0, 1), (0, 2), (2, 1)], [0.0, 0.0, 0.0]], [1.0, -1.0, 0.0]),
    ("an item without a rating is refused", [[(0, 3)], [0.0, 0.0]], {"raises": "ValueError"}),
    ("an item against itself is refused", [[(1, 1)], [0.0, 0.0]], {"raises": "ValueError"}),
]


def run_transfer(candidate, cases):
    observations = []
    for label, arguments, expected in cases:
        supplied = copy.deepcopy(arguments)
        try:
            actual = [round(x, 6) + 0.0 for x in candidate(*supplied)]
        except NotImplementedError:
            actual = {"unfinished": True}
        except Exception as error:
            actual = {"raises": type(error).__name__}
        passed = actual == expected and supplied == arguments
        observations.append(
            {"case": label, "expected": expected, "observed": actual, "passed": passed}
        )
        print("PASS" if passed else "NEEDS_WORK", label, "expected", expected, "observed", actual)
    return observations


transfer_observations = run_transfer(bt_gradient, TRANSFER_CASES)
TRANSFER_PASSED = all(row["passed"] for row in transfer_observations)
print("TRANSFER_STATUS", "PASS" if TRANSFER_PASSED else "NEEDS_WORK")

Design a counterexample and retrieve the mechanism

Add a case with an expected value you worked out by hand, and rerun the driver. Then fit ratings: start from zeros, repeatedly add a small multiple of your gradient, and re-center to mean zero. Use preferences sampled from ratings you choose, and check that the fit recovers them. Finally, explain what a preference grader that always favors the longer answer would teach a model trained against it.

Save your evidence and explain the result

Fill in the prediction notes and your explanation before saving. Include:

• the exact observed value, and the input that caused it;
• your code's invocation point;
• one failed hypothesis;
• the strongest claim the evidence still cannot support.

These simulations assume equally good candidates and independent cases. Real candidates differ, and when one is much better the curse is smaller; the chapter's measured prompt search is an example.

explanation_notes = {
    "causal_trace": "Explain the input, learner invocation and observed result.",
    "failed_hypothesis": "Describe a prediction the evidence changed.",
    "remaining_limit": "Name the guarantee not established by this experiment.",
}
course_submission = {
    "unit": "ch17-b",
    "planned_minutes": 90,
    "starting_evidence": globals().get("HANDOFF_ORIGIN", "INDEPENDENT_UNIT_A"),
    "prediction": prediction_notes,
    "explanation": explanation_notes,
    "core_report": exercise_report,
    "transfer": transfer_observations,
    "explanation_review": "HUMAN_REVIEW_REQUIRED",
}
submission_path = COURSE_WORK / "ch17-b-submission-v1.json"
submission_path.write_text(
    json.dumps(course_submission, indent=2, sort_keys=True), encoding="utf-8"
)
print("Saved evidence:", submission_path)
print(
    "COURSE_REPORT="
    + json.dumps(
        {
            "unit": "ch17-b",
            "transfer_passed": TRANSFER_PASSED,
            "starting_evidence": course_submission["starting_evidence"],
            "edition": "student",
        },
        sort_keys=True,
    )
)

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.