Controlled Improvement: Unit A Exercises

Student edition · 90 minutes of dedicated work · 2026-09-09

Runs on Google Colab as it ships today, or on any local Python 3.12+ kernel.

This is one of two practical units for Chapter 17. Unit A constructs and connects the mechanism; Unit B investigates a controlled failure, repairs it and transfers the invariant. Each is a complete ninety-minute session, with its own setup and required conceptual introductions. Basic Python variables, conditions, loops, functions, lists and dictionaries are the starting knowledge. Libraries and specialized concepts used here are introduced below before the main task.

By the end you should be able to:

1. Explain the chapter's mechanism using a prediction and an observed intermediate result.
2. Build activation of a staged immutable skill: require named positive cases, detect candidate mutation, compare the active configuration after evaluation and atomically activate the intended version.
3. Solve admit activation only against complete current evidence using changed inputs and an independent expectation.
4. Retain your implementation, failed/corrected observations, causal explanation and limits.

Installation is preparation time. These are planning estimates, not measured completion times. Use the reference primers when a term is unfamiliar; in Unit B retrieve an explanation before re-reading it. Run All checks that the artifact executes. Unfinished student functions deliberately produce NEEDS_WORK. Keep your first attempt before opening answers.

This notebook belongs to the construction edition. Its supplied teaching runtime is embedded, so it can run without the textbook or another notebook. Where code uses REFERENCE_LESSON, that is the frozen runtime exercise identifier; the reader-facing chapter and saved unit identifiers use the current edition. Building against a supplied runtime is not proof that you have constructed all of its dependencies.

Run the self-contained setup

Open this notebook in Google Colab with the badge above, or use a local Python 3.12+ Jupyter kernel, with Pydantic 2. If needed, run %pip install "pydantic==2.13.4" once in a separate cell and restart the kernel. Package installation needs internet; the lesson itself needs no repository download, API key or prior notebook. The closing extension can use an OpenAI key from Colab's Secrets pane to put a live model behind the same tools; without a key it replays a recorded transcript and says so.

The collapsed cell contains 90 frozen teaching files. Base85 represents compressed bytes as text; zlib decompresses them; SHA-256 checks that the decoded files match this edition. These are supplied packaging operations, not learner algorithms. tempfile creates an isolated working copy; Path handles file locations; sys.path tells Python where the supplied modules live; a .pth file, the mechanism an editable install uses, tells reviewed local subprocesses the same. The code is available for inspection below and performs no package installation itself. The subsequent lesson teaches the libraries used by the mechanisms you will implement.

Run setup on every fresh kernel. It writes scratch runtime files separately from your retained practical-work/ch17-a folder. Rerunning setup restores the frozen support files and keeps your saved work. Restarting a kernel clears variables, not saved submission files. Source basis: Sovereign Agent 444c5f6. Some tasks use reviewed local subprocesses; they are not an OS sandbox.

Supplied offline setup and teaching files

The complete supplied setup cell is in the downloadable notebook. Run it there before attempting the cells below. This web edition omits only that compressed setup payload.

Commit to a prediction before the examples

prediction_notes = {
    "prediction": "Write the expected behavior before running the worked example.",
    "reason": "Name the input and rule behind that prediction.",
    "falsifier": "Name an observation that would prove the explanation wrong.",
    "revision": "After execution, explain what changed in your understanding.",
}

Reading the Python vocabulary used in this notebook

You need basic assignments, if, loops, functions, lists and dictionaries. The less familiar features used by the supplied code are introduced here. A library is reusable code that Python can import. The standard library ships with Python; Pydantic is an additional package. An import makes a name available, but does not mean that you have completed the exercise.

JSON is text for exchanging structured values. A Python dictionary is an in-memory object; the JSON representation is a string. Use json.dumps to encode and json.loads to decode. Decoding proves that text has valid JSON syntax, not that its fields match our business contract. Predict which of the following two decoded objects could describe a stock count.

import json

intro_data = {"sku": "MANGO", "count": 3}
intro_text = json.dumps(intro_data, sort_keys=True)
print(type(intro_data).__name__, type(intro_text).__name__, intro_text)
print(json.loads(intro_text))
print("Also valid JSON:", json.loads('["not", "a", "stock", "record"]'))
assert json.loads(intro_text) == intro_data

The first result is a dictionary; the second is a list. Before indexing a decoded object, check the shape that your function promises to accept. An exception interrupts the normal path. raise ValueError(...) refuses an invalid value; try/except lets a caller inspect that expected refusal. Catch the expected class, rather than turning every programming error into apparent success. finally runs cleanup even when an earlier operation raises.

An annotation, such as count: int, documents the expected type. It does not by itself enforce the type at runtime. A class defines a kind of object; an instance holds one object's data. @dataclass asks Python to generate routine construction and comparison methods from annotated fields. frozen=True prevents ordinary reassignment of the instance's fields; it does not make every object nested inside those fields immutable. A method is a function attached to a class; self refers to the instance receiving the call.

from dataclasses import dataclass


@dataclass(frozen=True)
class IntroObservation:
    operation: str
    count: int


intro_observation = IntroObservation("count-mango", 3)
print(intro_observation.operation, intro_observation.count)
assert intro_observation == IntroObservation("count-mango", 3)

A callback is a function passed to another function. This is how the classroom harness invokes your implementation. The argument candidate below is a function object; parentheses perform the call. Predict the two answers before execution, then trace the result to the callback.

def intro_apply(candidate, value):
    return {"input": value, "observed": candidate(value)}


def intro_double(value):
    return value * 2


print(intro_apply(intro_double, 3))
print(intro_apply(lambda value: value + 2, 3))
assert intro_apply(intro_double, 3)["observed"] == 6

lambda value: value + 2 is a small anonymous function. A closure is a function that retains access to values from its surrounding scope. It can bind a tool to a shop snapshot. A shallow copy duplicates only the outer container; copy.deepcopy also copies nested containers used in these fixtures. A set stores distinct values; required <= allowed asks whether every required item is allowed. frozenset is the corresponding immutable set. A tuple groups ordered values; (value,) is a one-item tuple, including the comma.

Paths and cleanup. Path represents a filesystem location. path / "file.json" constructs a child path; read_text and write_text read and write text. A context manager, used with with, manages entry and exit. A temporary-directory context removes its contents on exit. Save your submission outside temporary runtime directories. Reopening a file is different from reusing a Python variable: the former tests retained bytes, while the latter only tests this kernel.

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as intro_folder:
    intro_path = Path(intro_folder) / "observation.json"
    intro_path.write_text(json.dumps(intro_data), encoding="utf-8")
    intro_reopened = json.loads(intro_path.read_text(encoding="utf-8"))
    assert intro_reopened == intro_data
    print("Read from a file:", intro_reopened)

Retrieval check: explain JSON versus a dictionary, annotation versus validation, class versus instance, and defining a callback versus invoking it. Change the callback above so an incorrect implementation visibly changes the observed output. This distinction will matter when grading your connected work. Reference: Python's JSON, dataclasses, and pathlib documentation.

Pydantic: turn an input dictionary into a checked object

Pydantic is an additional Python library for validating data. Its model is a class describing fields, not a neural network. Inherit from BaseModel, declare annotated fields, then call model_validate on incoming data. A field without a default is required. A field with a default can be omitted. The result is an instance whose values you read with dot notation.

from pydantic import BaseModel, ConfigDict, Field, ValidationError


class IntroCourseRequest(BaseModel):
    model_config = ConfigDict(strict=True, extra="forbid")
    name: str = Field(min_length=1)
    quantity: int = Field(gt=0, le=1000)
    note: str = ""


intro_request = IntroCourseRequest.model_validate({"name": "mango", "quantity": 4})
print(intro_request.name, intro_request.quantity, repr(intro_request.note))
assert intro_request.note == ""

The annotation says the field's type. Field supplies constraints: gt=0 means greater than zero, le=1000 means at most 1000, and min_length=1 excludes an empty name. ConfigDict sets model-wide behavior. strict=True rejects conversions for this integer field, including "4", 4.0 and True; extra="forbid" rejects undeclared keys. Pydantic can otherwise convert some compatible inputs, so choose this boundary deliberately rather than assuming every accepted input arrived in the expected type.

Predict which rule refuses each payload. ValidationError reports a failed contract. Its errors() entries contain loc, the field location, and type, the failure category. Catching that expected exception lets the notebook inspect the failure and continue.

intro_bad_requests = [
    {"name": "mango", "quantity": "4"},
    {"name": "mango", "quantity": True},
    {"name": "", "quantity": 4},
    {"name": "mango", "quantity": 0},
    {"name": "mango", "quantity": 4, "approved": True},
    {"quantity": 4},
]
for intro_bad_request in intro_bad_requests:
    try:
        IntroCourseRequest.model_validate(intro_bad_request)
    except ValidationError as intro_error:
        print([(item["loc"], item["type"]) for item in intro_error.errors(include_input=False)])
    else:
        raise AssertionError("An invalid input crossed the declared contract")

Use model_dump() for a Python dictionary, model_dump_json() for JSON text, and model_validate_json() to parse and validate JSON. model_json_schema() describes the contract; it is neither an instance's current values nor an invocation of the business handler.

intro_serialized = intro_request.model_dump_json()
intro_schema = IntroCourseRequest.model_json_schema()
assert IntroCourseRequest.model_validate_json(intro_serialized) == intro_request
print("Actual values:", intro_request.model_dump())
print("Quantity contract:", intro_schema["properties"]["quantity"])
assert intro_schema["properties"]["quantity"]["exclusiveMinimum"] == 0

Four is valid input to this schema even if the shop needs six. Pydantic checks the declared shape and constraints; the handler still needs authoritative stock, price and permission. Ordinary assignments to an existing instance are not automatically revalidated unless configured for assignment validation. This lesson validates new input at the boundary and uses the resulting values. Explain these limits before relying on a model object in a transaction or tool call.

Chapter 2's full introduction expands this pattern with a separate data-repair checkpoint. This notebook contains the required pattern here so prior Pydantic experience is not needed. References: models, fields, and strict mode.

SQLite from the first row to an atomic change

A dictionary disappears when its process ends. A database can retain records so a later process can resume from evidence. SQLite is an embedded database: Python's sqlite3 library opens a local database file without starting a separate database server. SQL is the language used to define, select and change its records. A table has named columns and rows. A primary key identifies a row; a query asks for rows satisfying a condition.

Start with a deliberately small preference table. Read the SQL as instructions: create the table, insert one named value, then select the value for one session. ? is a parameter placeholder; the values are passed separately so they are data, not SQL instructions. fetchone() returns one row or None; it does not guarantee that a matching row exists.

import sqlite3
from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as intro_sql_folder:
    intro_db_path = Path(intro_sql_folder) / "example.sqlite"
    intro_db = sqlite3.connect(intro_db_path, autocommit=True)
    intro_db.execute("CREATE TABLE preference (id INTEGER PRIMARY KEY, session TEXT, value TEXT)")
    intro_db.execute("INSERT INTO preference (session,value) VALUES (?,?)", ("lucy", "09:00"))
    intro_row = intro_db.execute(
        "SELECT value FROM preference WHERE session=?", ("lucy",)
    ).fetchone()
    print("Matching row:", intro_row)
    assert intro_row == ("09:00",)
    assert (
        intro_db.execute(
            "SELECT value FROM preference WHERE session=?", ("another-session",)
        ).fetchone()
        is None
    )
    intro_db.close()
    intro_reopen = sqlite3.connect(intro_db_path, autocommit=True)
    assert intro_reopen.execute("SELECT count(*) FROM preference").fetchone()[0] == 1
    intro_reopen.close()
print("The row survived closing and reopening its connection.")

Index [0] selects the first column of a returned tuple. sqlite3.Row is an alternative row factory that also permits named-column access. dict(row) then produces an ordinary dictionary. The book's Database wrapper supplies that configuration and its schema; the wrapper is course code, while sqlite3 is the standard library. You will use the public connection and transaction methods explained at the exercise boundary, rather than needing to reconstruct the wrapper.

Now consider a budget. Moving five cents from reserved to spent requires two values to change together. A transaction makes a group of local changes commit together or roll back together. The example explicitly controls SQL transactions with autocommit=True and SQL statements. BEGIN IMMEDIATE starts a write transaction; COMMIT keeps its changes; ROLLBACK discards them. Predict the row after the deliberately raised exception. Catching an error alone would not undo the first update; the rollback is the operation that restores the prior state.

intro_ledger = sqlite3.connect(":memory:", autocommit=True)
intro_ledger.execute(
    "CREATE TABLE budget (id INTEGER PRIMARY KEY, reserved INTEGER, spent INTEGER)"
)
intro_ledger.execute("INSERT INTO budget VALUES (1,5,0)")
intro_ledger.execute("BEGIN IMMEDIATE")
try:
    intro_ledger.execute("UPDATE budget SET reserved=0 WHERE id=1")
    raise ValueError("injected failure before the matching spend update")
except ValueError:
    intro_ledger.execute("ROLLBACK")
assert intro_ledger.execute("SELECT reserved,spent FROM budget").fetchone() == (5, 0)
intro_ledger.execute("BEGIN IMMEDIATE")
intro_ledger.execute("UPDATE budget SET reserved=0,spent=5 WHERE id=1")
intro_ledger.execute("COMMIT")
print(
    "After a complete change:", intro_ledger.execute("SELECT reserved,spent FROM budget").fetchone()
)
intro_ledger.close()

The literal :memory: creates a temporary database inside this connection; it is useful for the small experiment, but the earlier file example establishes persistence. Neither example proves that a remote supplier rolls back when the local transaction rolls back. An external operation has its own state and evidence.

SQL you will meet later. UPDATE ... SET ... WHERE ... changes selected rows. AND combines conditions. ORDER BY makes an ordering explicit; absent that clause, do not rely on row order. count(*) counts rows; sum(amount) totals a column and can be NULL on an empty input; coalesce(sum(amount),0) uses zero for that empty aggregate. A UNIQUE constraint rejects duplicate identities. GROUP BY status computes one aggregate per status.

An invariant is a condition that must remain true across operations, such as nonnegative reserved money. A snapshot is a consistent view at one point; two separate reads can describe different moments unless their transaction contract binds them. In the book, with db.immediate() groups related writes. It is a course-defined context manager with the commit/rollback purpose you just observed. Do not assume that an arbitrary with connection has identical behavior under every SQLite autocommit setting.

Your prediction: two workers both read ten remaining cents outside a transaction and each approve seven. Why can both believe the next order fits? Explain what must be checked together with the write. Then change the example's initial reserved amount and repeat the failure. Reference: Python's SQLite tutorial and transaction control.

Improvement changes a version under an evidence contract

Lucy notices that an opening procedure omits reserved stock. Editing the active procedure in place seems convenient, but destroys the relationship between an earlier evaluation and the version now running. This chapter separates candidate, evaluation and activation. A candidate is a proposed immutable version. Evaluation produces evidence for specified cases under a specified configuration. Activation changes which version future work will use.

A regression is behavior that used to satisfy a requirement and no longer does after a change. A regression suite retains earlier cases while adding the newly discovered failure. Passing only the new case can hide damage to an old one. Rollback restores a prior active version; it does not erase effects produced while the intervening version was active.

Record what was evaluated

A result named “PASS” without its candidate and baseline is ambiguous. Suppose evaluation starts with active version 2, but someone activates version 3 before evaluation finishes. The result may be valid for the old configuration and stale for the new one. Optimistic concurrency means doing expensive work without holding a long write transaction, then checking that the assumed baseline is still current before committing the change.

intro_active = {"opening": 2, "delivery": 1}
intro_evaluated_baseline = dict(intro_active)
intro_results = {"ordinary-stock": True, "reserved-stock": True}
intro_active["delivery"] = 2
intro_stale = intro_active != intro_evaluated_baseline
print("Cases passed:", all(value is True for value in intro_results.values()))
print("Configuration changed:", intro_stale)
assert intro_stale

The two observations can both be true: all named cases passed and activation must refuse because its baseline changed. Re-running against the current configuration is a different act from ignoring the stale flag. The comparison belongs inside the transaction that changes active rows, so another writer cannot slip between the final check and activation.

Exact truth values and complete case coverage

Python treats many values as truthy, including integer one and a nonempty string. A callback contract that requires exact boolean True should not accept those values as test evidence. Missing a required case is also different from an observed failure. Require named positive evidence for every required case before activation.

intro_required_cases = {"ordinary-stock", "reserved-stock"}
intro_reports = [
    {"ordinary-stock": True, "reserved-stock": True},
    {"ordinary-stock": True},
    {"ordinary-stock": True, "reserved-stock": 1},
    {"ordinary-stock": True, "reserved-stock": False},
]
for intro_report in intro_reports:
    intro_complete = intro_required_cases <= intro_report.keys()
    intro_positive = intro_complete and all(
        intro_report[name] is True for name in intro_required_cases
    )
    print(intro_report, "admissible evidence:", intro_positive)

Only the first report satisfies this evidence contract. This does not prove that the evaluator itself is a trustworthy oracle. It establishes that activation honors the declared evidence shape. The evaluation chapter supplies the separate reasoning about independent expected results.

Freeze both the candidate and the comparison baseline

A callback could mutate the candidate object while evaluating it. Serializing the candidate before evaluation and checking afterward detects this change. A shallow reference to the same mutable dictionary would not preserve the earlier bytes. Similarly, a version name without its content identity is too weak if content can change under that name.

The actual activate_skill path stages an immutable version, captures the active snapshot, evaluates outside the write transaction, then checks candidate and baseline inside the activation transaction. The context builder later reads active rows, making activation observable in the next model input. That final use is the connection: storing a candidate row alone does not change the running procedure.

Unit A constructs this complete boundary. Unit B removes the stale-configuration protection and uses an evaluation callback that deliberately changes another active skill. Inspect which version is active after the attempted activation and after reopening SQLite. The transfer introduces missing cases, False, integer one, candidate mutation and an uncontended valid activation.

Before coding, draw the sequence baseline capture → evaluation → transaction → baseline compare → active swap. Place a second writer at each gap and predict the outcome. Then explain what evidence should remain after a refusal and after a rollback. “Self-improvement” is not permission for the agent to bypass this version and evaluation boundary.

Understand the supplied execution interface

The course runtime is provided so your implementation can be connected to real callers and storage. SourceTask(ROOT, chapter) makes a private copy. install(source) replaces only the declared function; visible() invokes the real chapter probe; save(path, result) retains a successful implementation and its evidence. load(path) checks the saved identities and hashes. inject_failure() changes the declared boundary; repair(fragment) replaces that broken fragment. close() removes the scratch copy after you retain evidence. These methods are supplied harness operations, not additional packages you must discover or install.

RuntimeLab provides the same copied-source failure experiment without the complete-function construction layer. Its run method records exit status, observations and the compared expectation. A subprocess log from an unfinished learner implementation is feedback about that implementation; it is not a successful connection. A syntax error in the notebook cell itself is a separate issue to fix. The task below names which interface it uses.

For direct-function units, the visible driver calls your callback without installing a source string. In either case, trace where your code is invoked. Supplied fixtures, database wrappers and replay models are labeled infrastructure; your own implementation and changed-case explanation are the evidence of learning.

Main practical: construct, connect and challenge

Design question: A test result has a configuration boundary?

Build activation of a staged immutable skill: require named positive cases, detect candidate mutation, compare the active configuration after evaluation and atomically activate the intended version.

You edit a complete function in a temporary copy of src/sovereign_agent/assistant_context.py. The real callers, database and tool boundaries remain connected. The self-contained setup above supplies the frozen development runtime. The notebook runs reviewed local subprocesses; it is not a security sandbox. No live account is required.

Predict before running

Evaluate outside the write transaction, deliberately activate another skill inside the callback, then inspect the active snapshot.

Write the expected result and a falsifying observation before running. Include one legal action, one refusal, and the exact-empty case where the interface permits it. Explain the consequence for Lucy if your prediction is wrong.

import json
import os
import runpy
from pathlib import Path

ROOT = COURSE_ROOT

SourceTask = runpy.run_path(str(ROOT / "book/always_on/exercises/source_tasks_v1.py"))["SourceTask"]
REFERENCE_LESSON = 13
HANDOFF = Path("ch17-unit-a-handoff-v1.json")

Construct the complete mechanism

Implement activate_skill in the string below. Keep the named signature and existing helper interfaces. The starter is intentionally incomplete; its failed connection is reported separately from whether the notebook itself executed. A constant answer cannot stand in for the real mechanism.

Inspect the supplied caller around the function in src/sovereign_agent/assistant_context.py. Draw the data path from the observed output through this function to its actual input or query. Then write your implementation from the contract.

implementation_source = """
def activate_skill(
    db: Database,
    name: str,
    version: str,
    *,
    evaluate: Callable[[Skill], dict[str, bool]],
    required_cases: frozenset[str],
    expected_state: str | None = None,
) -> dict[str, bool]:
    raise NotImplementedError("Construct this chapter mechanism")
"""

Hint 1: the design

A passing callback can describe an obsolete configuration.

Hint 2: the boundary

Inspect the parameters and the caller in src/sovereign_agent/assistant_context.py. Identify validation, durable state and the first externally observable effect. Preserve the existing surrounding helper contracts.

Hint 3: the structure

Capture a baseline and serialized candidate before evaluating; require exact True values; compare the snapshot inside db.immediate before swapping active rows.

Connect to the cumulative runtime

The following installs your complete function into the copied runtime and executes the chapter probe against it. It saves your implementation and the observed connection for Unit B only after that connection succeeds.

def connect_build(source):
    task = SourceTask(ROOT, REFERENCE_LESSON)
    try:
        task.install(source)
        result = task.visible()
        if result["status"] == "PASS":
            task.save(HANDOFF, result)
        return result
    finally:
        task.close()


build_result = connect_build(implementation_source)
print("CONNECTION", build_result["status"])
print("OBSERVATION", build_result["observation"])

Challenge and transfer

Try missing cases, False and integer one, candidate mutation, explicit stale state and a concurrent activation. Preserve the valid uncontended case.

Use a fresh SourceTask, install your implementation and edit only its copied probe to run the changed input. Keep the expected result in your prediction notes, independent of your implementation. Call task.run("MY_TRANSFER", expected=your_expected) and close the task in finally. Retain both a valid and a refused case so rejecting everything cannot pass.

The instructor runs additional cases with different identities and boundaries against the real source. Passing the visible connection alone is not the transfer verdict. Do not put instructor solutions or holdouts into a student submission.

Save and explain

After success, submit ch14-unit-a-handoff-v1.json, your source, prediction notes and changed-input observations. Unit B checks the chapter, runtime hash and exact implementation hash and re-executes your code.

Explain which input or state caused the output, which observation would refute the explanation and what remains outside the guarantee: Authored callbacks exercise state control; they are not a substitute for live regression evidence.

exercise_report = {
    "unit": "ch17-a",
    "attempted": 1,
    "completed": int(build_result["status"] == "PASS"),
    "failed": int(build_result["status"] != "PASS"),
    "skipped": 0,
    "connection": build_result["status"],
    "handoff": "WRITTEN" if build_result["status"] == "PASS" else "NOT_READY",
}
print("EXERCISE_REPORT=" + json.dumps(exercise_report, sort_keys=True))

Changed-constraint construction: Admit activation only against complete current evidence

Allow twenty minutes. Spend three minutes predicting, ten implementing and tracing, five on a new case of your own, and two explaining the surviving limitation. This is dedicated work, not an invitation to run a supplied answer. Both units revisit the same invariant after different core experiences; in Unit B, attempt this task from memory before consulting Unit A.

Implement transfer_check(before, current, results, required). Return True only if required is nonempty, all required case names exist in results with value exactly True, and the active configuration dictionaries before/current are equal. Do not mutate any input. Extra observed cases are permitted but do not replace required cases.

Write your expected values before running the table. Keep one accepted case and one refusal. Your function is passed directly into the driver below. The driver copies inputs and checks they remain unchanged; it does not replace your implementation with the reference answer.

Hint 1: identify the authoritative inputs Name the source field for each output value. Which input changes while the rule remains the same?

Hint 2: choose the boundary cases Start with exact empty, exact equality and one value on each side of the boundary where valid. Do not add a special case for a visible product name or operation identity.

def transfer_check(before, current, results, required):
    raise NotImplementedError("Bind evaluation coverage to the current configuration")

import copy
import json

TRANSFER_CASES = [
    (
        "current positive",
        [
            {"opening": 2},
            {"opening": 2},
            {"normal": True, "reserved": True},
            ["normal", "reserved"],
        ],
        True,
    ),
    ("stale baseline", [{"opening": 2}, {"opening": 3}, {"normal": True}, ["normal"]], False),
    ("missing case", [{}, {}, {"normal": True}, ["normal", "reserved"]], False),
    ("truthy integer", [{}, {}, {"normal": 1}, ["normal"]], False),
]


def same_transfer_value(actual, expected):
    if type(actual) is not type(expected):
        return False
    if isinstance(expected, dict):
        return actual.keys() == expected.keys() and all(
            same_transfer_value(actual[key], value) for key, value in expected.items()
        )
    if isinstance(expected, list):
        return len(actual) == len(expected) and all(
            same_transfer_value(a, e) for a, e in zip(actual, expected, strict=True)
        )
    return actual == expected


def run_transfer(candidate, cases):
    observations = []
    for label, arguments, expected in cases:
        supplied = copy.deepcopy(arguments)
        before = copy.deepcopy(supplied)
        raised = None
        try:
            actual = candidate(*supplied)
        except NotImplementedError:
            raised = "NotImplementedError"
            actual = {"unfinished": True}
        except Exception as error:
            raised = type(error).__name__
            actual = {"raises": raised}
        expects_error = isinstance(expected, dict) and set(expected) == {"raises"}
        correct = (
            raised == expected["raises"]
            if expects_error
            else (raised is None and same_transfer_value(actual, expected))
        )
        passed = correct and same_transfer_value(supplied, before)
        observations.append(
            {"case": label, "expected": expected, "observed": actual, "passed": passed}
        )
        print("PASS" if passed else "NEEDS_WORK", label, "expected", expected, "observed", actual)
    return observations


transfer_observations = run_transfer(transfer_check, TRANSFER_CASES)
TRANSFER_PASSED = all(row["passed"] for row in transfer_observations)
print("TRANSFER_STATUS", "PASS" if TRANSFER_PASSED else "NEEDS_WORK")

Design a counterexample and retrieve the mechanism

Add one new case with an independently calculated expected outcome to TRANSFER_CASES and rerun the driver. Change one condition at a time. Then deliberately replace your candidate with a constant answer in a temporary copy and show a case that rejects it. Restore your implementation. Explain why that counterexample is stronger than repeating the original example with a new name.

Without viewing the worked example, write the invariant in words and trace one observed value back to its input. Identify which part is a local fixture result and which claim would need a live provider, host or external-system observation. Keep a first attempt even if you used a hint.

Save your evidence and explain the result

Fill the prediction notes and your explanation before saving. Include the exact observed value, the input or retained row that caused it, your code's invocation point, one failed hypothesis, and the strongest claim the evidence still cannot support. A completed code cell alone does not earn explanation credit. Do not label reference-start behavior as your own Unit A construction.

Keep this edited notebook, the Markdown if used for notes, saved handoff files, and the JSON record below. Your work folder survives scratch cleanup and can be reopened in a new kernel. An instructor can ask for an unseen case after the visible checks; keep your implementation general.

explanation_notes = {
    "causal_trace": "Explain the input, learner invocation and observed result.",
    "failed_hypothesis": "Describe a prediction the evidence changed.",
    "remaining_limit": "Name the guarantee not established by this experiment.",
}
course_submission = {
    "unit": "ch17-a",
    "planned_minutes": 90,
    "starting_evidence": globals().get("HANDOFF_ORIGIN", "INDEPENDENT_UNIT_A"),
    "prediction": prediction_notes,
    "explanation": explanation_notes,
    "core_report": exercise_report,
    "transfer": transfer_observations,
    "explanation_review": "HUMAN_REVIEW_REQUIRED",
}
submission_path = COURSE_WORK / "ch17-a-submission-v1.json"
submission_path.write_text(
    json.dumps(course_submission, indent=2, sort_keys=True), encoding="utf-8"
)
print("Saved evidence:", submission_path)
print(
    "COURSE_REPORT="
    + json.dumps(
        {
            "unit": "ch17-a",
            "transfer_passed": TRANSFER_PASSED,
            "starting_evidence": course_submission["starting_evidence"],
            "edition": "student",
        },
        sort_keys=True,
    )
)

Extension: let a model propose the next version of a procedure

Every activation above was requested by you. activate_skill exists because in production the request comes from somewhere less careful: a model that has read Lucy's message and wants to change how the assistant behaves. The boundary this chapter built is an evidence contract. A version is staged as immutable bytes, evaluated outside the write transaction, admitted only when every required case is exactly True, and switched on inside db.immediate() after the active configuration is compared with the baseline captured before evaluation.

This section puts a model behind that contract through three tools. list_skills reads the assistant_skills table and the snapshot digest. stage_skill writes a TOML file and calls the runtime's stage_skill. activate_skill calls the runtime's activate_skill with the shop's own regression suite from reference_organizations.store.evaluation: eight authored scenarios run through the offline shop model, plus one check on the candidate's wording. The model can propose a version; it cannot pick the cases, mark one as passed, or skip the snapshot comparison. Those live in the handler and in the runtime, not in the prompt.

Prediction: the next cell prints the three schemas and the required case names. Write down which two tools are consequential, which argument is a list, and how many required cases the offline suite contributes before you run it.

import json
from pathlib import Path

from pydantic import BaseModel, ConfigDict, Field

from reference_organizations.store.agent import OfflineShopModel
from reference_organizations.store.evaluation import CASES, candidate_checks, evaluate
from sovereign_agent.agent_loop import Limits, run_loop
from sovereign_agent.assistant_context import activate_skill, context, skill_snapshot, stage_skill
from sovereign_agent.database import Database
from sovereign_agent.llm_lab import ScriptedModel, describe, scripted_turn, tool_trace
from sovereign_agent.tool_dispatch import Dispatcher, ExecutableTool

LLM_SKILL_DB = COURSE_WORK / "ch17-llm-skills.sqlite"
SKILL_DIR = COURSE_WORK / "ch17-llm-skills"
SKILL_NAMES = frozenset({"opening", "reporting"})
REQUIRED_CASES = frozenset({f"{case.name}:0" for case in CASES} | {"instructions_name_gbp"})


class NoArguments(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)


class SkillVersion(NoArguments):
    name: str = Field(pattern=r"^[a-z][a-z0-9_-]{0,63}$")
    version: str = Field(pattern=r"^[a-zA-Z0-9._-]{1,64}$")


class SkillCandidate(SkillVersion):
    instructions: str = Field(min_length=1, max_length=8192)
    requires: list[str] = Field(default_factory=list, max_length=16)


def fresh_skill_database(path):
    """Start from an empty skill table so reruns and replays observe the same rows."""
    for stale in (path, Path(f"{path}-wal"), Path(f"{path}-shm"), path.with_suffix(".authority")):
        stale.unlink(missing_ok=True)
    SKILL_DIR.mkdir(exist_ok=True)
    for toml_file in SKILL_DIR.glob("*.toml"):
        toml_file.unlink()
    return Database(path)


def regression_suite(db):
    """The shop's eight offline scenarios plus Lucy's wording rule, keyed by case name."""

    def evaluate_candidate(skill):
        report = evaluate(OfflineShopModel, skill=skill, skills=skill_snapshot(db)[1])
        checks = candidate_checks(report)
        checks["instructions_name_gbp"] = "USD" in skill.instructions
        return checks

    return evaluate_candidate


def skill_desk(db, evaluator):
    """Three tools over the real runtime; the ledger keeps the reason of every refusal."""
    refusals = []

    def list_skills(_):
        digest, _active = skill_snapshot(db)
        rows = db.connection.execute(
            "SELECT name,version,active FROM assistant_skills ORDER BY name,version"
        ).fetchall()
        return {"configuration": digest[:12], "skills": [dict(row) for row in rows]}

    def stage_candidate(args):
        path = SKILL_DIR / f"{args.name}-{args.version}.toml"
        lines = [
            "name = " + json.dumps(args.name),
            "version = " + json.dumps(args.version),
            "instructions = " + json.dumps(args.instructions),
            "requires = " + json.dumps(args.requires),
        ]
        path.write_text("\n".join(lines) + "\n", encoding="utf-8")
        try:
            skill = stage_skill(db, path)
        except ValueError as error:
            refusals.append(("stage_skill", args.name, args.version, str(error)))
            raise
        return {"staged": skill.name, "version": skill.version, "active": False}

    def activate_candidate(args):
        try:
            results = activate_skill(
                db, args.name, args.version, evaluate=evaluator, required_cases=REQUIRED_CASES
            )
        except (ValueError, PermissionError) as error:
            refusals.append(("activate_skill", args.name, args.version, str(error)))
            raise
        return {"activated": args.name, "version": args.version, "cases": results}

    def before_write(call):
        if call.arguments.get("name") not in SKILL_NAMES:
            refusals.append((call.name, call.arguments.get("name"), None, "name not in catalog"))
            raise PermissionError("name not in catalog")

    dispatcher = Dispatcher(
        [
            ExecutableTool(
                "list_skills",
                "List staged skill versions and which are active.",
                NoArguments,
                list_skills,
            ),
            ExecutableTool(
                "stage_skill",
                "Stage one immutable skill version. Staging never activates it.",
                SkillCandidate,
                stage_candidate,
                consequential=True,
            ),
            ExecutableTool(
                "activate_skill",
                "Run the shop's regression suite on a staged version and activate it only if "
                "every required case passes.",
                SkillVersion,
                activate_candidate,
                consequential=True,
            ),
        ],
        allowed=frozenset({"list_skills", "stage_skill", "activate_skill"}),
        before_write=before_write,
    )
    return dispatcher, refusals


skill_db = fresh_skill_database(LLM_SKILL_DB)
skill_dispatcher, skill_refusals = skill_desk(skill_db, regression_suite(skill_db))
for tool_schema in skill_dispatcher.schemas():
    function = tool_schema["function"]
    print(function["name"], "requires", function["parameters"].get("required", []))
    print("   ", function["description"])
print("Required cases:", sorted(REQUIRED_CASES))
SKILL_MESSAGES = [
    {
        "role": "system",
        "content": (
            "You maintain the procedures of Lucy's ice cream shop assistant. A procedure is a "
            "versioned skill: stage_skill stores a version, activate_skill runs the shop's "
            "regression suite and switches the active version only when every required case "
            "passes. Lucy's rule: any procedure that mentions prices must say USD. Read the "
            "current skills first. Finish with a short plain-words summary of which version is "
            "active and why any activation was refused."
        ),
    },
    {
        "role": "user",
        "content": (
            "Please add an 'opening' procedure for the morning. Version 1 wording: 'Greet the "
            "customer, check stock before promising anything, and quote prices in pounds.' "
            "If the shop refuses it, revise the wording to say USD and stage it as version 2."
        ),
    },
]

The schemas are the model_json_schema() output of the two Pydantic classes above, wrapped in the {"type": "function", ...} envelope. The name and version patterns are the same ones Skill enforces, so a version like ../etc is refused as invalid_arguments before any file is written. The before_write gate adds Lucy's own rule on top: only names in her catalog may be staged or activated. A schema refusal and a catalog refusal are different boundaries and the ledger records which one fired.

Run the loop against a recorded transcript

The turns below are what a model replied, written down. Turn one lists the empty table. Turn two stages version 1 and asks to activate it. Turn three stages version 2 and asks again. Turn four lists the table to confirm. Turn five is the answer, which ends the loop.

Prediction: which activation is refused, and by which check? The offline shop model does not read the skill's wording, so all eight scenario cases pass for both versions; the instructions_name_gbp case is what separates them. Recall that Dispatcher.invoke flattens the ValueError to tool_failed; the model never sees the reason, only the ledger does.

VERSION_ONE = (
    "Greet the customer, check stock before promising anything, and quote prices in pounds."
)
VERSION_TWO = "Greet the customer, check stock before promising anything, and quote prices in USD."
RECORDED_TURNS = [
    scripted_turn(calls=[{"name": "list_skills", "arguments": {}}]),
    scripted_turn(
        calls=[
            {
                "name": "stage_skill",
                "arguments": {"name": "opening", "version": "1", "instructions": VERSION_ONE},
            },
            {"name": "activate_skill", "arguments": {"name": "opening", "version": "1"}},
        ]
    ),
    scripted_turn(
        calls=[
            {
                "name": "stage_skill",
                "arguments": {"name": "opening", "version": "2", "instructions": VERSION_TWO},
            },
            {"name": "activate_skill", "arguments": {"name": "opening", "version": "2"}},
        ]
    ),
    scripted_turn(calls=[{"name": "list_skills", "arguments": {}}]),
    scripted_turn(
        "Version 1 of 'opening' was staged but its activation was refused, so it stays inactive. "
        "Version 2 says USD, passed the regression suite and is now the active opening procedure."
    ),
]
LOOP_LIMITS = Limits(model_calls=8, tool_calls=16, seconds=120, output_tokens=4096)

scripted_result = run_loop(
    ScriptedModel(RECORDED_TURNS), skill_dispatcher, SKILL_MESSAGES, limits=LOOP_LIMITS
)
print(describe(scripted_result, "scripted"))
print("Refusal ledger:", skill_refusals)
scripted_trace = tool_trace(scripted_result)
assert scripted_result.status == "COMPLETED"
assert [step["ok"] for step in scripted_trace] == [True, True, False, True, True, True]
assert scripted_trace[2]["result"] == "tool_failed"
assert skill_refusals == [
    ("activate_skill", "opening", "1", "candidate did not pass all required regression cases")
]
assert scripted_trace[4]["result"]["cases"]["instructions_name_gbp"] is True
assert len(scripted_trace[4]["result"]["cases"]) == len(REQUIRED_CASES)
# The durable state: one active version, chosen by evidence, and one activation event.
active_after = [(skill.name, skill.version) for skill in skill_snapshot(skill_db)[1]]
assert active_after == [("opening", "2")]
activated_events = skill_db.connection.execute(
    "SELECT count(*) FROM events WHERE kind='assistant.skill.activated'"
).fetchone()[0]
assert activated_events == 1
guidance = context(skill_db, "lucy", "Brief me", allowed=frozenset({"list_stock"}))
assert VERSION_TWO in guidance[0]["content"] and VERSION_ONE not in guidance[0]["content"]
# Every observation in the transcript carries the identifier of the request it answers.
assert all(m.get("tool_call_id") for m in scripted_result.messages if m["role"] == "tool")

The refused activation is an ordinary failed observation; the loop carries on and the model adapts on the next turn. The reason is only in the ledger, the tool_failed is only in the transcript, and the assistant_skills table holds both versions with exactly one marked active. context() now injects version 2 as guidance, and guidance is not permission: the dispatcher allowlist did not change.

Run the same loop against a live model

Nothing changes except the model and a fresh database. build_model looks for an OPENAI_API_KEY in Colab's Secrets pane (the key icon in the left sidebar: add a secret with that name and switch on notebook access for it), then in the process environment for a local kernel. Without a key it returns the scripted model again and the record says so.

The live model is gpt-5.1 through the openai library, which Colab ships preinstalled; on a local kernel run %pip install openai once. The model may write version 2 first, stage both before activating either, or try a name outside the catalog. What it cannot do is activate a version that fails a required case, or anything under a name Lucy did not list.

Prediction: write down what a careful assistant should stage and activate from Lucy's message, then run the cell and compare it with the model's judgment, or with the recorded transcript when the run is scripted.

from sovereign_agent.llm_lab import DEFAULT_MODEL, build_model, resolve_api_key

live_db = fresh_skill_database(LLM_SKILL_DB)
live_dispatcher, live_refusals = skill_desk(live_db, regression_suite(live_db))
chosen_model = build_model(RECORDED_TURNS, model=DEFAULT_MODEL)
print("Model source:", chosen_model.source, "| key found:", resolve_api_key() is not None)
live_result = run_loop(chosen_model, live_dispatcher, SKILL_MESSAGES, limits=LOOP_LIMITS)
print(describe(live_result, chosen_model.source))
print("Refusal ledger:", live_refusals)
live_trace = tool_trace(live_result)
assert live_result.status in {"COMPLETED", "MODEL_CALL_LIMIT", "TOOL_LIMIT", "MODEL_FAILED"}
# Whatever the model did, every active skill passed the suite and sits inside the catalog.
live_active = skill_snapshot(live_db)[1]
for skill in live_active:
    assert skill.name in SKILL_NAMES
    live_checks = regression_suite(live_db)(skill)
    assert all(live_checks[case] is True for case in REQUIRED_CASES), (skill.name, live_checks)
staged_names = {row[0] for row in live_db.connection.execute("SELECT name FROM assistant_skills")}
assert staged_names <= SKILL_NAMES
# One activation event per activation the dispatcher reported as ok, and no others.
ok_activations = [s for s in live_trace if s["tool"] == "activate_skill" and s["ok"]]
live_events = live_db.connection.execute(
    "SELECT count(*) FROM events WHERE kind='assistant.skill.activated'"
).fetchone()[0]
assert live_events == len(ok_activations)
assert all(m.get("tool_call_id") for m in live_result.messages if m["role"] == "tool")

Challenge: a passing evaluation that describes an obsolete configuration

This is the Unit B failure, run through the loop instead of the probe. The next cell stages two versions of reporting and wraps the regression suite so that, while a candidate is being evaluated, the other reporting version is activated. Every evaluation still returns the same case results; only the active configuration moves underneath it. The recorded transcript does not change.

Predict before running: version 1 is still refused, but is it refused for the same reason? Read the order of checks in activate_skill: the case check runs before the snapshot comparison, so version 1 fails on instructions_name_gbp first. Version 2 passes every case and is then refused inside db.immediate() with active skill configuration changed during evaluation. The scripted answer still claims version 2 is active. The ledger and the table say otherwise, which is the point: prose is not evidence.

In your notes, separate what the scripted run proves (the contract, the gate, the ledger, the rows) from what only a live run can show (whether the model reads the table after a refusal instead of repeating itself). The saved record keeps the two apart.

stale_db = fresh_skill_database(LLM_SKILL_DB)
plain_suite = regression_suite(stale_db)
for reporting_version in ("1", "2"):
    reporting_path = SKILL_DIR / f"reporting-{reporting_version}.toml"
    reporting_path.write_text(
        f'name = "reporting"\nversion = "{reporting_version}"\n'
        'instructions = "Report totals in USD."\nrequires = []\n',
        encoding="utf-8",
    )
    stage_skill(stale_db, reporting_path)


def concurrent_suite(skill):
    """Activate the other reporting version while this candidate is under evaluation."""
    active = {s.name: s.version for s in skill_snapshot(stale_db)[1]}
    other = "2" if active.get("reporting") == "1" else "1"
    activate_skill(
        stale_db, "reporting", other, evaluate=plain_suite, required_cases=REQUIRED_CASES
    )
    return plain_suite(skill)


stale_dispatcher, stale_refusals = skill_desk(stale_db, concurrent_suite)
stale_result = run_loop(
    ScriptedModel(RECORDED_TURNS), stale_dispatcher, SKILL_MESSAGES, limits=LOOP_LIMITS
)
print(describe(stale_result, "scripted"))
print("Refusal ledger:", stale_refusals)
assert stale_result.status == "COMPLETED"
assert [reason for _, _, _, reason in stale_refusals] == [
    "candidate did not pass all required regression cases",
    "active skill configuration changed during evaluation",
]
assert [(s.name, s.version) for s in skill_snapshot(stale_db)[1]] == [("reporting", "2")]

llm_lab_report = {
    "unit": "ch17-a",
    "scripted": {
        "status": scripted_result.status,
        "tool_calls": scripted_result.tool_calls,
        "refused": [list(entry) for entry in skill_refusals],
        "active": active_after,
    },
    "session": {
        "source": chosen_model.source,
        "model": DEFAULT_MODEL if chosen_model.source == "live" else None,
        "status": live_result.status,
        "model_calls": live_result.model_calls,
        "tool_calls": live_result.tool_calls,
        "refused": [list(entry) for entry in live_refusals],
        "active": [(skill.name, skill.version) for skill in live_active],
        "answer": live_result.answer[:400],
    },
    "stale_configuration": {
        "refused": [list(entry) for entry in stale_refusals],
        "active": [(skill.name, skill.version) for skill in skill_snapshot(stale_db)[1]],
    },
}
llm_report_path = COURSE_WORK / "ch17-a-llm-lab-report-v1.json"
llm_report_path.write_text(json.dumps(llm_lab_report, indent=2, sort_keys=True), encoding="utf-8")
print("Saved evidence:", llm_report_path)
print("LLM_LAB_REPORT=" + json.dumps(llm_lab_report["session"], sort_keys=True))
for open_db in (skill_db, live_db, stale_db):
    open_db.close()

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.