First Model Call: Unit B Exercises

Student edition · 90 minutes of dedicated work · 2026-09-26

Runs on Google Colab as it ships today (Python 3.12), or on any local Python 3.12+ kernel. It needs nothing beyond Python's standard library.

This is the second of Chapter 1's two practical units. It starts from Unit A's tokenizer.

A colleague has compared tokenizers for Lucy's model and chosen the one with the lowest perplexity. The choice is wrong, and the reason is one of the most common mistakes in measuring language models: comparing per-token numbers across tokenizers, on training text. You will reproduce the mistake, build the fair measurement, and apply it to a real model's log-probabilities.

By the end you should be able to:

Define cross-entropy and perplexity, and explain why next-token training minimizes cross-entropy.
Explain why per-token perplexities of models with different tokenizers cannot be compared, and why training text cannot rank models.
Implement bits per character, and use it on held-out text to choose correctly.
Compute bits per character from a real model's natural-log log-probabilities.

Minutes	Dedicated work	Saved evidence
0–10	Predict which tokenizer the colleague should have chosen	Written prediction
10–30	Likelihood, cross-entropy and the colleague's comparison	Worked outputs and handoff
30–60	Construct `bits_per_character` and pass the visible cases	Learner code and grade table
60–70	Choose a tokenizer fairly, including Unit A's	Measured table
70–85	Changed-constraint task: bits per character from a real model	Transfer results
85–90	Explain the result and save evidence	Retained submission

These times are planning estimates, not measured completion times. Run All only checks that the notebook executes; the unfinished student functions deliberately report NEEDS_WORK. Keep your first attempt before you open the answers.

Run the self-contained setup

The unit needs only Python's standard library. The collapsed cell supplies Lucy's notes, the byte-pair encoder, the bigram model, and a correct softmax: everything Unit A built or used.

Run setup on every fresh kernel. Your saved work lives in practical-work/ch01-b.

Supplied setup: corpus, tokenizer, model and softmax

The complete supplied setup cell is in the downloadable notebook. Run it there before attempting the cells below. This web edition omits only that compressed setup payload.

Commit to a prediction before the examples

Your colleague trained the same bigram model on Lucy's notes with four tokenizers: 0, 50, 100 and 200 requested merges (the last stops at 134, when no pair repeats). They chose the tokenizer whose model had the lowest perplexity per token on the training notes.

Before you run anything, write down:

which tokenizer you expect that rule to choose;
whether you expect the same tokenizer to be best on notes the model never saw.

python

prediction_notes = {
    "prediction": "Write which tokenizer the colleague's rule picks, and which wins on new notes.",
    "reason": "Name the rule behind that prediction.",
    "falsifier": "Name an observation that would prove the explanation wrong.",
    "revision": "After execution, explain what changed in your understanding.",
}

Likelihood and cross-entropy

A model assigns a probability to a whole text by multiplying, token by token, the probability it gave each token after the one before:

P(t_1, \dots, t_n) = \prod_{k=2}^{n} P(t_k \mid t_{k-1}).

The average negative base-2 logarithm of those probabilities is the model's cross-entropy on the text: the bits it needs per token.

\mathcal{H} = -\frac{1}{n-1} \sum_{k=2}^{n} \log_2 P(t_k \mid t_{k-1}).

Perplexity is $2^{\mathcal{H}}$ , the number of equally likely tokens the model is, on average, torn between. Training a language model means minimizing cross-entropy on its training text. For a bigram model, counting is that minimization, done exactly.

The colleague's comparison, reproduced

The cell below implements the colleague's rule: per-token perplexity on the training notes. Predict its choice before you run it.

python

def colleague_perplexity(model, tokens):
    index = {token: i for i, token in enumerate(model.vocabulary)}
    bits = 0.0
    for previous, following in zip(tokens, tokens[1:], strict=False):
        bits -= math.log2(softmax(model.logits(previous))[index[following]])
    return 2 ** (bits / (len(tokens) - 1))


intro_rows = []
for requested in (0, 50, 100, 200):
    intro_merges = train_bpe(SHOP_NOTES, requested)
    intro_tokens = encode(SHOP_NOTES, intro_merges)
    intro_model = BigramModel(intro_tokens, intro_merges)
    intro_rows.append(
        (len(intro_merges), round(colleague_perplexity(intro_model, intro_tokens), 3))
    )
for merges_used, perplexity in intro_rows:
    print(merges_used, "merges: training perplexity per token", perplexity)
print("colleague chooses", min(intro_rows, key=lambda row: row[1])[0], "merges")

Two things are wrong with this comparison.

Per-token numbers compare different tasks. With no merges, a token is one character; with 134, it averages 3.2 characters. Predicting a bigger unit is harder per prediction and easier per character, so per-token perplexity rewards or punishes the tokenizer, not the model. The fair unit is bits per character: total bits divided by the length of the text.
Training text rewards memorization. A model can always fit its own training text better. The last merges turn whole phrases of the notes into single tokens, which helps on those notes and nowhere else. Only held-out text ranks models honestly.

Choose an explicit starting point

This unit starts from Unit A's tokenizer. To use your own, replace None with the path to your practical-work/ch01-a/ch01-unit-a-handoff-v1.json. Leave it as None to start from the supplied reference. Your submission records which you chose.

python

LEARNER_HANDOFF = None

python

import shutil

COURSE_INPUT = COURSE_WORK / "ch01-unit-a-handoff-v1.json"
if LEARNER_HANDOFF is not None:
    learner_input = Path(LEARNER_HANDOFF).expanduser().resolve()
    if not learner_input.is_file():
        raise FileNotFoundError("The selected learner handoff does not exist")
    if learner_input != COURSE_INPUT.resolve():
        shutil.copy2(learner_input, COURSE_INPUT)
    HANDOFF_ORIGIN = "LEARNER_SELECTED"
else:
    reference_handoff = {
        "unit": "ch01-a",
        "status": "COMPLETED",
        "merges": [[a.hex(), b.hex()] for a, b in train_bpe(SHOP_NOTES, 100)],
        "alpha": 0.001,
        "entropy_table": [],
    }
    COURSE_INPUT.write_text(json.dumps(reference_handoff, indent=2, sort_keys=True) + "\n")
    HANDOFF_ORIGIN = "SUPPLIED_REFERENCE"
print("Starting evidence:", HANDOFF_ORIGIN)

python

handoff = json.loads(COURSE_INPUT.read_text(encoding="utf-8"))
handoff_status = "VERIFIED" if handoff.get("status") == "COMPLETED" else "INVALID"
unit_a_merges = [(bytes.fromhex(a), bytes.fromhex(b)) for a, b in handoff["merges"]]
unit_a_alpha = handoff["alpha"]
print("UNIT_A_HANDOFF", handoff_status)
print("Unit A tokenizer:", len(unit_a_merges), "merges, alpha", unit_a_alpha)

Main practical: construct, connect and challenge

1. Construct `bits_per_character`

bits_per_character(model, merges, text) must:

encode text with merges;
sum $-\log_2 P(t_k \mid t_{k-1})$ over every consecutive pair of tokens, using the model's logits and the supplied softmax;
divide the total by the number of characters in text, not the number of tokens;
raise ValueError when the text encodes to fewer than two tokens, because there is nothing to predict.

The starter below computes the colleague's quantity instead. Run the visible cases, then repair it.

python

def bits_per_character(model, merges, text):
    """Average bits the model needs per character of text."""
    tokens = encode(text, merges)
    index = {token: i for i, token in enumerate(model.vocabulary)}
    bits = 0.0
    for previous, following in zip(tokens, tokens[1:], strict=False):
        bits -= math.log2(softmax(model.logits(previous))[index[following]])
    return bits / (len(tokens) - 1)

Hint 1: the denominator

The numerator is the same total number of bits either way. Only the denominator changes: len(text) characters, not len(tokens) - 1 predictions.

Hint 2: the edge case

A single token has no successor, so there are no predictions and no bits. Refuse it with ValueError rather than dividing zero bits by a length and reporting a perfect score.

python

import copy


class FixedModel:
    """A stub model with the same logits after every token: answers you can work out by hand."""

    def __init__(self, vocabulary, logits):
        self.vocabulary, self._logits = tuple(vocabulary), list(logits)

    def logits(self, previous):
        return list(self._logits)


def outcome(call):
    try:
        return round(call(), 6)
    except Exception as error:
        return type(error).__name__


AB = FixedModel([b"a", b"b"], [0.0, 0.0])
SKEW = FixedModel([b"a", b"b"], [math.log(3), 0.0])
MERGED = FixedModel([b"a", b"b", b"ab"], [0.0, 0.0, 0.0])
VISIBLE_CASES = [
    ("fair coin, four characters", lambda: bits_per_character(AB, [], "abab"), 0.75),
    ("a likely then unlikely token", lambda: bits_per_character(SKEW, [], "ab"), 1.0),
    (
        "merged tokens, characters not tokens",
        lambda: bits_per_character(MERGED, [(b"a", b"b")], "abab"),
        round(math.log2(3) / 4, 6),
    ),
    ("one token predicts nothing", lambda: bits_per_character(AB, [], "a"), "ValueError"),
]


def grade(cases):
    rows = []
    for label, call, expected in cases:
        observed = outcome(call)
        rows.append(
            {
                "case": label,
                "expected": expected,
                "observed": observed,
                "status": "PASS" if observed == expected else "FAIL",
            }
        )
    return rows


visible_results = grade(VISIBLE_CASES)
VISIBLE_PASSED = all(row["status"] == "PASS" for row in visible_results)
for visible_row in visible_results:
    print(visible_row["status"], visible_row["case"], "->", visible_row["observed"])
print("VISIBLE_CONTRACT", "PASSED" if VISIBLE_PASSED else "NEEDS_WORK")

2. Connect: choose a tokenizer fairly, including Unit A's

Once the visible cases pass, the cell below scores every candidate tokenizer, including the one from your Unit A handoff. For each it reports bits per character on the training notes and on the held-out notes, and it chooses by held-out bits per character.

Predict the choice, and predict which candidate the training column would have chosen.

python

connected = None
if VISIBLE_PASSED:
    candidates = {f"{n} requested merges": train_bpe(SHOP_NOTES, n) for n in (0, 50, 200)}
    candidates["Unit A tokenizer"] = unit_a_merges
    table = []
    for name, merges in candidates.items():
        model = BigramModel(encode(SHOP_NOTES, merges), merges, alpha=unit_a_alpha)
        table.append(
            {
                "tokenizer": name,
                "merges": len(merges),
                "train_bits_per_character": round(bits_per_character(model, merges, SHOP_NOTES), 3),
                "held_out_bits_per_character": round(
                    bits_per_character(model, merges, HELD_OUT_NOTES), 3
                ),
            }
        )
    fair = min(table, key=lambda row: row["held_out_bits_per_character"])
    by_training = min(table, key=lambda row: row["train_bits_per_character"])
    connected = {
        "table": table,
        "fair_choice": fair["tokenizer"],
        "training_choice": by_training["tokenizer"],
    }
    for row in table:
        print(row)
    print("choose by held-out bits per character:", fair["tokenizer"])
    print("training bits per character would choose:", by_training["tokenizer"])
else:
    print("CONNECTION_NOT_READY — repair bits_per_character, then run again.")

Trace the two choices. Which row does training favor, and by how much does it lose on held-out notes? Where did the extra merges' advantage on training text go? Write the one sentence you would send your colleague.

Exit ticket

Answer three questions:

Why can two models with different tokenizers not be compared by per-token perplexity?
Why does the training column always favor the tokenizer with the most merges?
What would you need, beyond three held-out notes, before trusting the choice for Lucy's real model?

python

exercise_report = {
    "unit": "ch01-b",
    "attempted": 1,
    "completed": int(VISIBLE_PASSED),
    "failed": int(not VISIBLE_PASSED),
    "skipped": 0,
    "connection": "PASSED" if connected else "NOT_READY",
    "handoff": handoff_status,
}
print("EXERCISE_REPORT=" + json.dumps(exercise_report, sort_keys=True))

Changed-constraint construction: bits per character from a real model

Allow fifteen minutes: three to predict, eight to implement and check, and four for a case of your own.

A real model's API can return, for every token it generated or scored, the natural logarithm of its probability (logprobs). You have no access to its weights, only these numbers and the text. Compute the same fair measure from them.

Implement transfer_bits_per_character(log_probabilities, text):

log_probabilities is a list of natural-log probabilities, one per scored token;
return $-\sum \ell_k / \ln 2$ divided by the number of characters in text;
raise ValueError for an empty list, or for any value above 0 (a probability above one);
do not change the input.

The driver compares your result, rounded to six places, with values worked out by hand.

Hint: units Natural logs measure nats. Divide by $\ln 2 \approx 0.6931$ to get bits, then divide by characters. A log-probability of exactly 0 is allowed: probability one.

python

def transfer_bits_per_character(log_probabilities, text):
    raise NotImplementedError("Convert nats to bits, then divide by characters")

python

TRANSFER_CASES = [
    ("one certain token", [[0.0], "ok"], 0.0),
    ("two coin flips over four characters", [[math.log(0.5), math.log(0.5)], "abcd"], 0.5),
    ("a quarter-probability token", [[math.log(0.25)], "xy"], 1.0),
    ("an empty list is refused", [[], "text"], {"raises": "ValueError"}),
    ("a positive log-probability is refused", [[0.1], "text"], {"raises": "ValueError"}),
]


def run_transfer(candidate, cases):
    observations = []
    for label, arguments, expected in cases:
        supplied = copy.deepcopy(arguments)
        try:
            actual = round(candidate(*supplied), 6) + 0.0
        except NotImplementedError:
            actual = {"unfinished": True}
        except Exception as error:
            actual = {"raises": type(error).__name__}
        passed = actual == expected and supplied == arguments
        observations.append(
            {"case": label, "expected": expected, "observed": actual, "passed": passed}
        )
        print("PASS" if passed else "NEEDS_WORK", label, "expected", expected, "observed", actual)
    return observations


transfer_observations = run_transfer(transfer_bits_per_character, TRANSFER_CASES)
TRANSFER_PASSED = all(row["passed"] for row in transfer_observations)
print("TRANSFER_STATUS", "PASS" if TRANSFER_PASSED else "NEEDS_WORK")

Design a counterexample and retrieve the mechanism

Add a case with an expected value you worked out by hand, and rerun the driver. Then take the chapter's live receipt: 49 answer tokens with a total log-probability of −49.9 nats, and an answer of about 250 characters. Compute its bits per character, and compare it with the bigram model's held-out figure. Explain why that comparison is still not fair: the texts differ.

Save your evidence and explain the result

Fill in the prediction notes and your explanation before saving. Include:

the exact observed value, and the input that caused it;
your code's invocation point;
one failed hypothesis;
the strongest claim the evidence still cannot support.

This unit ranked tokenizers on three held-out notes. It did not measure how much that ranking would vary with a different held-out sample.

python

explanation_notes = {
    "causal_trace": "Explain the input, learner invocation and observed result.",
    "failed_hypothesis": "Describe a prediction the evidence changed.",
    "remaining_limit": "Name the guarantee not established by this experiment.",
}
course_submission = {
    "unit": "ch01-b",
    "planned_minutes": 90,
    "starting_evidence": globals().get("HANDOFF_ORIGIN", "INDEPENDENT_UNIT_A"),
    "prediction": prediction_notes,
    "explanation": explanation_notes,
    "core_report": exercise_report,
    "transfer": transfer_observations,
    "explanation_review": "HUMAN_REVIEW_REQUIRED",
}
submission_path = COURSE_WORK / "ch01-b-submission-v1.json"
submission_path.write_text(
    json.dumps(course_submission, indent=2, sort_keys=True), encoding="utf-8"
)
print("Saved evidence:", submission_path)
print(
    "COURSE_REPORT="
    + json.dumps(
        {
            "unit": "ch01-b",
            "transfer_passed": TRANSFER_PASSED,
            "starting_evidence": course_submission["starting_evidence"],
            "edition": "student",
        },
        sort_keys=True,
    )
)

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.