In-context learning: why a skill's exact words must be tested

Draft chapter. The textbook guide covers setup and the supplied code. Practice in Exercise Book 7, then check Solutions 7 once you have tried it.

Lucy wants the same useful opening brief tomorrow, even when her message is only “Prepare the replenishment drafts.” In Chapters 3 and 5, we kept the procedure in a Python message: inspect current stock, use each product's calculated need, create the appropriate drafts, and report their amounts in USD. That procedure is now worth naming. It should be possible to review and revise it without changing the tool dispatcher or copying instructions into every channel adapter.

Moving the words into a file is easy. Deciding when those words become active is the engineering problem. A plausible revision might stop creating drafts and merely describe them. Another might demand a purchasing tool that this stage of the agent does not have. A third might pass an evaluation while someone changes the active configuration underneath it. A useful skill implementation makes each of those situations visible.

Part A first measures why that matters: a skill is words in the model's context, and equally reasonable wordings and example orders change the answers. Part B then builds the machinery. In it you will extract the opening procedure into a local TOML file, validate and stage an immutable version, evaluate it, and activate it only against the configuration you tested. You will then select the skill into the bounded context from Chapter 5. The model still uses the loop and dispatcher you built. Loading a procedure neither adds a Python function nor changes the authority to invoke one.

Learning objectives

Part A treats a prompt as a random variable. After it you should be able to:

• explain in-context learning as conditioning, with no change to the model;
• measure a prompt's score as one draw from a distribution of equivalent prompts, and compare its spread with case-sampling noise;
• show that few-shot examples, and their order, can help, do nothing, or hurt;
• compare two prompts with a paired test, and explain why the winner must be re-measured.

Part B builds versioned skills. After it you should be able to:

• distinguish a tool, a skill, a workflow and a policy; implement bounded local skill loading; preserve the identity of a staged version; separate staging, evaluation, activation and eligibility; and prove that an active skill reaches the model without granting additional tool authority.

The deliverable is an opening-check procedure whose version survives a process restart and whose activation has an observable evaluation result. You will reproduce failures involving changed version content, an incomplete evaluation, a missing required tool and a changed configuration. The default exercises use authored model responses. An optional local-model run measures a few actual decisions; it does not turn a small case suite into a general guarantee about the model.

Part A: a prompt is one draw from a distribution

A skill is instructions, and perhaps examples, placed in the model's context. Nothing about the model changes: the same weights compute a different next-token distribution because the tokens before it differ, as Chapter 1 showed. This is in-context learning. It makes a skill cheap to write and cheap to change. It also means that small, meaning-preserving changes to the words can change the answers.

This part measures how much. Treat the prompt as a random variable. There are many equally reasonable ways to write the same instruction and to order the same examples, and the one you picked is one draw. Its score is one sample from a distribution of scores.

The functions live in the chapter's learner file.

import json
import runpy

sens = runpy.run_path(
    "book/textbook/learner/profrod_sovereign_agent_ch07_prompt_sensitivity_learner.py"
)
measured = json.loads(
    open("docs/evidence/book-ch07/ch07-prompt-sensitivity-receipt-v1.json").read()
)

The experiment

The chapter's experiment asks a model to classify twenty-four of Lucy's requests with one of four labels:

• stock: report a stock level;
• draft: prepare an unsent replenishment draft;
• clarify: the product is missing or unclear;
• refuse: the request would buy, pay or send an order.

Every request is asked under twenty-four prompts. There are six paraphrases of the same instruction, all stating the same four rules. Each is asked with no examples, and with the same four labeled examples in three different orders. Everything else is held fixed: the model, temperature zero, and the cases.

uv run python \
    book/textbook/experiments/profrod_sovereign_agent_textbook_ch07_prompt_sensitivity_v1.py \
    --out ch07-prompt-sensitivity-receipt.json

One run, recorded on 2026-09-26 with Ollama 0.32.5 on macOS (arm64), took two and a half minutes for 1,152 answers.

Measured: the spread across equivalent prompts

for m in measured["models"]:
    zero, few = m["zero_shot_paraphrases"], m["few_shot_prompts"]
    print(
        f"{m['model']:13} no examples: mean {zero['mean']:.3f}, range {zero['min']:.3f}-{zero['max']:.3f}"
        f" | with examples: mean {few['mean']:.3f}, range {few['min']:.3f}-{few['max']:.3f}"
    )
    print(
        f"{'':13} case-sampling sd {m['case_sampling_sd_at_mean']}, "
        f"prompt sd {zero['sd']} (paraphrases), {few['sd']} (with examples)"
    )

qwen2.5:0.5b  no examples: mean 0.389, range 0.250-0.458 | with examples: mean 0.498, range 0.375-0.625
              case-sampling sd 0.102, prompt sd 0.082 (paraphrases), 0.075 (with examples)
qwen2.5:1.5b  no examples: mean 0.590, range 0.375-0.708 | with examples: mean 0.567, range 0.375-0.750
              case-sampling sd 0.101, prompt sd 0.125 (paraphrases), 0.111 (with examples)

Six instructions that say the same thing produce accuracies from 0.375 to 0.708 on the larger model. Across prompts, the standard deviation is 0.08 to 0.13. That is as large as the spread you would expect from drawing a fresh set of twenty-four cases, about 0.10. The choice of wording is as big a source of variation as the choice of test cases. A score from one prompt says as much about the prompt as about the model.

Examples help one model and not the other, and their order matters

Few-shot examples are the textbook remedy. Here they raised the smaller model's mean accuracy by eleven points and left the larger model's slightly lower. The order of the same four examples mattered as much as including them.

for m in measured["models"]:
    print(
        f"{m['model']:13} range across three orders, per paraphrase: {m['order_range_per_paraphrase']}"
    )

qwen2.5:0.5b  range across three orders, per paraphrase: [0.208, 0.167, 0.125, 0.041, 0.041, 0.084]
qwen2.5:1.5b  range across three orders, per paraphrase: [0.208, 0.292, 0.167, 0.291, 0.167, 0.167]

View full-size diagram

xychart-beta
    title "qwen2.5:1.5b: six equivalent instructions"
    x-axis "Wording of the instruction" ["1", "2", "3", "4", "5", "6"]
    y-axis "Accuracy on 24 requests" 0 --> 1
    bar "No examples" [0.708, 0.583, 0.625, 0.708, 0.542, 0.375]
    bar "Four examples, one order" [0.75, 0.75, 0.667, 0.708, 0.667, 0.542]
    bar "Same examples, another order" [0.542, 0.458, 0.5, 0.417, 0.5, 0.375]

On the larger model, reordering four examples moved one paraphrase's accuracy by 29 points. Models are sensitive to which example comes last, among other things, and this sensitivity is not something a reader of the prompt can see.

Most cases do not have a stable answer

for m in measured["models"]:
    duel = m["best_vs_worst"]
    print(
        f"{m['model']:13} all 24 prompts agree on {m['all_prompts_agree_share']:.0%} of cases; "
        f"best {m['best']} vs worst {m['worst']}: {duel['only_first']}-{duel['only_second']}, "
        f"McNemar p = {duel['mcnemar_p']}"
    )

qwen2.5:0.5b  all 24 prompts agree on 8% of cases; best p1-order_b vs worst p2-zero: 12-3, McNemar p = 0.0352
qwen2.5:1.5b  all 24 prompts agree on 29% of cases; best p1-order_b vs worst p5-order_c: 9-0, McNemar p = 0.00391

For the smaller model, all twenty-four prompts gave the same verdict on only two of twenty-four cases. The best and worst prompts differ by more than noise, by Chapter 16's paired test. But choosing the best of twenty-four prompts on these same cases would be Chapter 17's winner's curse. The chosen prompt's score must be measured again on cases it was not chosen on.

Following the format is part of what a prompt controls

Every prompt told the model to answer with one of four words. Some answers did not.

for model in ("qwen2.5:0.5b", "qwen2.5:1.5b"):
    runs = [r for r in measured["runs"] if r["model"] == model]
    unlabeled = [r for r in runs if r["label"] is None]
    common = {}
    for r in unlabeled:
        word = r["answer"].strip().lower().rstrip(".")
        common[word] = common.get(word, 0) + 1
    top = sorted(common.items(), key=lambda kv: -kv[1])[:3]
    print(f"{model:13} {len(unlabeled)} of {len(runs)} answers had no label; most common: {top}")

qwen2.5:0.5b  66 of 576 answers had no label; most common: [('order', 32), ('confirm', 10), ('report', 7)]
qwen2.5:1.5b  65 of 576 answers had no label; most common: [('report', 31), ('pay', 24), ('send', 5)]

About one answer in nine names the request's own verb instead of a label: "pay" for a payment request, "report" for a stock question. These are scored wrong, and deliberately so. The prompt asked for one of four labels, and a program that routes on the label cannot use "pay". A grader that accepted synonyms would have measured something else. As Chapter 5 found, a grader's rules are part of the measurement, and they must be stated.

What this means for a skill

Three rules follow, and the rest of this chapter builds them into the skill machinery.

• The exact bytes of a skill are the thing evaluated. Rewording or reordering is a new candidate, not an edit. This is why skills here are immutable, versioned, and evaluated before activation.
• Evaluate a skill across its reasonable variants when you can. Report the spread, not only one score.
• Compare skills on the same cases with a paired test, and confirm the winner on new cases.

Part B: reuse a tested opening procedure

Part A showed that the words of a procedure are part of its behavior. This part turns the opening procedure into a versioned skill whose exact bytes are staged, evaluated and activated.

Decide which part of the behavior should be data

A tool is executable code with an argument contract. The stock tool reads authoritative records; the draft tool checks quantity and calculates money. A skill is guidance describing how to use capabilities that already exist. A workflow fixes some sequencing in code, while a policy decides whether an operation is permitted. These four things can cooperate, but replacing one with another changes which component must enforce correctness.

For this exact replenishment calculation, a plain function is a strong alternative. The model is useful when Lucy also asks questions, combines a procedure with preferences or needs an explanation. The skill lets us study that flexible interface. It does not make the arithmetic more accurate than a function. Keep that comparison available: Chapter 16 will measure the added cost of the agent against a scripted baseline.

A skill file may say “only buy from an approved supplier.” That sentence expresses useful intent, but the purchase boundary must still check supplier authority. A skill may say “call list_stock first.” That is a procedural request which the model can violate; our transcript checks should detect the violation. Distinguishing these statements prevents a common mistake: counting a sentence in a prompt as an implemented control.

View full-size diagram

flowchart LR
    F[Local TOML file] --> V[Validate and stage]
    V --> E[Evaluate candidate]
    E --> A[Activate a version]
    A --> C[Select eligible context]
    C --> L[Owned model loop]
    L --> D[Existing tool dispatcher]
    P[Runtime authority] --> D

Our first format deliberately contains one procedure per file. There is no registry lookup, package installation, script execution or template expansion. TOML supplies readable strings and lists using Python's standard library. The installed runtime still has only Pydantic as its direct runtime dependency. A future folder containing examples and references would need explicit rules for which files are loaded, how they are bounded and which exact content an evaluation covered.

Write the opening skill

Create profrod_sovereign_agent_textbook_opening_check_v1.toml in your scratch directory. Its name identifies the procedure; its version identifies immutable content within that name. The requirements list declares tool names needed to make the procedure relevant. Instructions explain the actual sequence. The example uses the same shop vocabulary and stock calculation as the earlier chapters, so extracting the procedure does not reset the reader's running agent.

name = "opening_check"
version = "1"
requires = ["list_stock", "supplier", "draft_order"]
instructions = """
Prepare a morning replenishment brief. Read list_stock first. For each product,
use its needed field: this already accounts for reserved and incoming stock.
For every needed value greater than zero, call draft_order with that exact
SKU and quantity. Do not draft products whose needed value is zero. A verbal
recommendation does not replace creating the draft through the tool. Finish by
reporting each successful draft's quantity and total_cents in USD. A draft is
not a purchase. If a tool refuses a draft, read current stock before continuing.
"""

The important verb is “call.” A model can write a fluent answer saying Lucy needs six tubs without creating a draft through the tool. We observed that failure while constructing the earlier chapters. The revised procedure makes the expected action explicit, and the checker looks for successful tool observations. Neither the instruction alone nor the final sentence “done” establishes that the action occurred.

Notice what the file does not encode: the fixture's six-tub and four-tub answers. Those are consequences of current stock and thresholds. Embedding them would turn a reusable procedure into a memorized test answer. The procedure consumes the tool's needed field, which already accounts for reserved and incoming stock. That keeps the business calculation in the deterministic component where you can test it independently.

import hashlib
import json
import os
import runpy
import sqlite3
import stat
import tempfile
import tomllib
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from pydantic import BaseModel, ConfigDict, Field, ValidationError

MEMORY = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch05_memory_learner.py")
LOOP = runpy.run_path("book/textbook/learner/profrod_sovereign_agent_ch03_agent_loop_learner.py")
TOOLS = LOOP["shop_tools"]
record_event, ModelTurn, ToolCall = MEMORY["record_event"], LOOP["ModelTurn"], LOOP["ToolCall"]


class Skill(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    name: str = Field(pattern=r"^[a-z][a-z0-9_-]{0,63}$")
    version: str = Field(pattern=r"^[a-zA-Z0-9._-]{1,64}$")
    instructions: str = Field(min_length=1, max_length=8192)
    requires: list[str] = Field(default_factory=list, max_length=16)


source = Path("book/textbook/skills/profrod_sovereign_agent_textbook_opening_check_v1.toml")
skill = Skill.model_validate(tomllib.loads(source.read_text()))
print(skill.name, skill.version)
print(skill.requires)

opening_check 1
['list_stock', 'supplier', 'draft_order']

The name pattern keeps identities short and predictable. The version is a label, not a promise of semantic-version ordering. Version 2 does not become active simply because its string looks newer. Strict validation rejects unknown fields and incompatible types; a spelling mistake such as instruction should fail at admission rather than quietly produce an empty procedure.

The character limit on instructions and the byte limit on the whole file solve different problems. A non-ASCII character can occupy several bytes. TOML also includes field names, quoting and comments beyond the instruction string. We will enforce both limits at their respective boundaries. The requirements list is bounded in length, but declaring a name does not prove that the corresponding tool exists or is allowed.

try:
    Skill.model_validate({"name": "opening", "version": "1", "instruction": "Read stock"})
except ValidationError as error:
    print(sorted(item["type"] for item in error.errors()))

['extra_forbidden', 'missing']

Bound the file read itself

A preliminary path.stat().st_size check is insufficient. Between checking a path and reading it, another process can replace the file or enlarge it. The original implementation rejected oversized content after calling read_bytes(), but a controlled experiment made it read two million bytes before reporting the error. Rejection protected the database while failing to bound the read allocation.

The fix is to open once, inspect the opened descriptor, and read at most the allowed bytes plus one. The extra byte distinguishes an exactly full file from a file that has crossed the limit. On our macOS and Linux path, O_NOFOLLOW refuses a final symlink and O_NONBLOCK prevents opening a FIFO from waiting for a writer. The regular-file check applies to the opened object, not an earlier observation of its pathname.

def read_skill(path: Path) -> bytes:
    """Read one bounded regular file; POSIX flags refuse symlinks and FIFO waits."""
    if path.is_symlink():
        raise ValueError("bounded regular local skill file required")
    flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0)
    try:
        descriptor = os.open(path, flags)
        with os.fdopen(descriptor, "rb") as stream:
            observed = os.fstat(stream.fileno())
            if not stat.S_ISREG(observed.st_mode) or observed.st_size > 16_384:
                raise ValueError("bounded regular local skill file required")
            raw = stream.read(16_385)
    except OSError as error:
        raise ValueError("readable regular local skill file required") from error
    if len(raw) > 16_384:
        raise ValueError("skill changed beyond byte limit")
    return raw


print(read_skill(source) == source.read_bytes())

True

Opening once also clarifies identity during a rename. If the pathname is replaced after open, the descriptor still refers to the selected file. A concurrent in-place writer can still alter that file's contents; this helper is not a filesystem snapshot or a lock. We validate and hash the exact bounded bytes returned by the read, then persist that version. Authors should finish writing a candidate before requesting staging.

The optional flag lookup keeps the function importable elsewhere, but the no-follow and nonblocking guarantees here are demonstrated on POSIX hosts. Parent directories are operator-owned; this is not a sandbox for an attacker-controlled filesystem tree. Do not infer protection against every path race from one flag. Chapter 15 will place code execution behind a separate environment boundary.

scratch = tempfile.TemporaryDirectory(prefix="lucy-skill-chapter-")
root = Path(scratch.name)
bounded = root / "bounded.toml"
bounded.write_bytes(b"x" * 16_384)
print("exact limit:", len(read_skill(bounded)))
bounded.write_bytes(b"x" * 16_385)
try:
    read_skill(bounded)
except ValueError:
    print("oversize: refused")
link = root / "link.toml"
link.symlink_to(source.resolve())
try:
    read_skill(link)
except ValueError:
    print("symlink: refused")

exact limit: 16384
oversize: refused
symlink: refused

These probes test reading, not TOML validity: a file of repeated x bytes is not a skill. Admission is a sequence of checks. Passing the resource boundary permits parsing; passing parsing permits schema validation; passing the schema permits immutable staging. Keeping those steps separate lets an error explain which contract was violated.

Stage a version without activating it

We need a table whose primary key is (name, version) and whose partial unique index permits at most one active version of each name. The content column stores the validated model's JSON representation. The source column stores a SHA-256 digest of the raw TOML bytes, allowing a reviewer to identify the first file staged for that version. Staging defaults to inactive.

skill_schema = """
CREATE TABLE assistant_skills (
    name TEXT NOT NULL, version TEXT NOT NULL, content TEXT NOT NULL,
    source TEXT NOT NULL, active INTEGER NOT NULL DEFAULT 0,
    PRIMARY KEY(name, version)
);
CREATE UNIQUE INDEX assistant_skill_current
ON assistant_skills(name) WHERE active = 1;
"""
probe = sqlite3.connect(":memory:")
probe.executescript(skill_schema)
print([row[1] for row in probe.execute("PRAGMA table_info(assistant_skills)")])
probe.close()

['name', 'version', 'content', 'source', 'active']

The agent keeps this table in the store you built in Chapter 4, beside Chapter 5's memory: open_skills opens Chapter 5's memory store and applies the table as one more line of migrations, owned by skills. The independent in-memory connection above is only a schema experiment. There is no second production skill store. Every subsequent operation uses the same store and its immediate transaction contract.

SKILL_MIGRATIONS = {
    1: (
        "CREATE TABLE assistant_skills ("
        " name TEXT NOT NULL, version TEXT NOT NULL, content TEXT NOT NULL,"
        " source TEXT NOT NULL, active INTEGER NOT NULL DEFAULT 0,"
        " PRIMARY KEY (name, version))",
        # At most one active version of each name, whoever writes the table.
        "CREATE UNIQUE INDEX assistant_skill_current ON assistant_skills (name) WHERE active = 1",
    ),
}


def open_skills(path: str | Path):
    """Chapter 5's memory store, with the skill table as its own line of versions."""
    db = MEMORY["open_memory"](path)
    db.migrate("skills", SKILL_MIGRATIONS)
    return db


def stage_skill(db, path: Path) -> Skill:
    """Store one validated version, inactive. The same version with other content is refused."""
    raw = read_skill(path)
    skill = Skill.model_validate(tomllib.loads(raw.decode()))
    content = skill.model_dump_json()
    with db.immediate() as connection:
        old = connection.execute(
            "SELECT content FROM assistant_skills WHERE name=? AND version=?",
            (skill.name, skill.version),
        ).fetchone()
        if old and old[0] != content:
            raise ValueError("skill versions are immutable; stage a new version")
        connection.execute(
            "INSERT OR IGNORE INTO assistant_skills(name,version,content,source) VALUES (?,?,?,?)",
            (skill.name, skill.version, content, hashlib.sha256(raw).hexdigest()),
        )
    return skill


db_path = root / "agent.sqlite"
db = open_skills(db_path)
stage_skill(db, source)
stage_skill(db, source)
print(tuple(db.connection.execute("SELECT count(*),sum(active) FROM assistant_skills").fetchone()))

(1, 0)

The transaction makes the existing-content check and insertion one operation. A repeated identical candidate is harmless. Different validated content under the same identity is refused. This is an application contract enforced by this staging boundary; a program with unrestricted database-write access can violate application conventions. Model tools will not receive such access.

JSON content identity is slightly different from file identity. Adding a TOML comment can leave the validated content unchanged, in which case restaging is accepted and the original source digest remains. Changing instructions or the requirements list changes the content and requires a new version. The runtime does not reinterpret versions according to filenames, modification times or lexicographic order.

Failure experiment: overwrite yesterday's version

Suppose you edit the opening procedure to “Summarize stock only” and leave version = "1". If staging silently replaced the row, an earlier evaluation report for version 1 would now name different content. The version label would have lost its value as evidence. Instead, try the edit and observe the refusal.

changed = root / "opening-check-changed.toml"
changed.write_text('name="opening_check"\nversion="1"\ninstructions="Summarize stock only"\n')
try:
    stage_skill(db, changed)
except ValueError as error:
    print(error)
print(db.connection.execute("SELECT count(*) FROM assistant_skills").fetchone()[0])

skill versions are immutable; stage a new version
1

A proposed change should get a new version and remain inactive while it is examined. The operator may author that file directly or ask a model to propose its contents. Proposal does not imply installation: the model does not receive a tool that can activate its own procedure. Chapter 17 will develop diagnosis, comparison reports and rollback around this same boundary.

At this stage a version remains available even after another version becomes active. That history is useful for explaining a change and re-evaluating an older procedure. It is not an unlimited-storage guarantee: the file and instruction bounds limit one candidate, not the lifetime number of stored versions. A retention policy must distinguish inactive historical candidates from the version currently serving requests.

Bind evaluation to the active configuration

Evaluating one candidate in isolation can miss interactions with another active skill. The runtime therefore takes a snapshot of all active guidance, ordered by name, including each source digest. A hash identifies that configuration. The snapshot is obtained in one database query so it does not combine rows from different observations.

def skill_snapshot(db) -> tuple[str, tuple[Skill, ...]]:
    """One read binds active guidance and provenance to an evaluation baseline."""
    rows = [
        tuple(row)
        for row in db.connection.execute(
            "SELECT name,version,content,source FROM assistant_skills WHERE active=1 ORDER BY name"
        )
    ]
    digest = hashlib.sha256(json.dumps(rows).encode()).hexdigest()
    return digest, tuple(Skill.model_validate_json(row[2]) for row in rows)


before, active = skill_snapshot(db)
print("active procedures:", len(active))
print("snapshot digest length:", len(before))

active procedures: 0
snapshot digest length: 64

The digest is an identity check, not a quality score. Two configurations with different hashes may behave identically, while one unchanged configuration can produce different live-model responses. We use the hash for a narrow question: is this still the configuration against which the candidate was evaluated? Outcome evaluation answers a different question about the observed behavior on named cases.

Long model calls should not hold a database write lock. We load the immutable candidate, capture the baseline, run the evaluator outside the transaction, and then enter a short transaction to compare the active state and activate. If another activation changed the baseline, our result is stale. Retrying means evaluating against the new configuration, rather than applying an old result to a different one.

View full-size diagram

sequenceDiagram
    participant A as Activation request
    participant E as Evaluator
    participant D as SQLite
    A->>D: Read candidate and active-state hash
    A->>E: Evaluate immutable candidate
    E-->>A: Named case results
    A->>D: Begin immediate transaction
    A->>D: Compare current active-state hash
    alt Baseline unchanged and cases pass
        A->>D: Switch active version and record event
    else Baseline changed
        A->>D: Refuse activation
    end

def activate_skill(
    db,
    name: str,
    version: str,
    *,
    evaluate: Callable[[Skill], dict[str, bool]],
    required_cases: frozenset[str],
    expected_state: str | None = None,
) -> dict[str, bool]:
    """Evaluate outside any transaction, then activate only if every required case passed and
    the active configuration is still the one the evaluation started from."""
    row = db.connection.execute(
        "SELECT content FROM assistant_skills WHERE name=? AND version=?", (name, version)
    ).fetchone()
    if row is None or not required_cases:
        raise ValueError("staged skill and a nonempty regression suite required")
    skill = Skill.model_validate_json(row[0])
    baseline = skill_snapshot(db)[0] if expected_state is None else expected_state
    results = evaluate(skill)
    if skill.model_dump_json() != row[0]:
        raise ValueError(
            "evaluation changed the candidate instead of testing its immutable version"
        )
    if not required_cases.issubset(results) or any(value is not True for value in results.values()):
        raise ValueError("candidate did not pass all required regression cases")
    with db.immediate() as connection:
        if skill_snapshot(db)[0] != baseline:
            raise PermissionError("active skill configuration changed during evaluation")
        connection.execute("UPDATE assistant_skills SET active=0 WHERE name=?", (name,))
        connection.execute(
            "UPDATE assistant_skills SET active=1 WHERE name=? AND version=?", (name, version)
        )
        record_event(
            connection,
            "assistant.skill.activated",
            {"name": name, "version": version, "cases": results},
        )
    return results


try:
    activate_skill(
        db,
        "opening_check",
        "1",
        evaluate=lambda candidate: {"opening": True},
        required_cases=frozenset({"opening", "at_threshold"}),
    )
except ValueError as error:
    print(error)
print("active procedures:", len(skill_snapshot(db)[1]))

candidate did not pass all required regression cases
active procedures: 0

The activation function requires every named case and rejects any returned value that is not literally True. It also rejects an evaluator that mutates the candidate object while claiming to test its immutable version. Those checks protect the evidence boundary. They cannot determine whether a dishonest evaluator fabricated its booleans. The evaluator's actual work and expected answers must be inspected and tested too.

The callback above deliberately tests missing-case refusal; it is not a behavioral evaluation. For the real candidate, use authored scenarios and inspect actual tool observations. Keep live session preferences out of isolated evaluation data unless the scenario explicitly supplies them. Otherwise a private preference or an old generated answer could contaminate a supposedly reproducible case.

Evaluate the opening procedure on three visible cases

Our first suite covers an ordinary shortage, a product exactly at its threshold, and reserved stock that creates a shortage despite apparently sufficient physical stock. The answers are written with the fixtures. They are not generated by the agent being evaluated. The threshold case catches the tempting but incorrect rule “always draft something,” while the reserved-stock case catches a procedure that ignores committed stock.

The evaluator below is small on purpose. Each case gets its own tools over its own stock, through your Chapter 2 dispatcher. The candidate's guidance goes into the system message through skill_items, which keeps a skill only if every tool it requires is allowed and the whole item fits the budget, exactly as it will at work, and your Chapter 3 loop runs the model. The case passes only if the loop completed, every tool call succeeded, the successful drafts are exactly the authored ones, and an answer that reports drafts names USD. A fluent answer without the drafts fails, and so does a run that reaches the right drafts after a refused one. Chapter 16 builds the full harness, with more cases, a scripted baseline and reports.

You already constructed the loop and transcript checker in Chapter 3; evaluation reuses that observable boundary across several inputs. The offline model follows an inspectable policy and tests the wiring. It does not read the skill and learn how to behave, so an offline pass alone is insufficient evidence that a live model will follow a revision. The checkpoint also feeds the evaluator three near misses, a partial draft, prose without drafts and a refused tool call, and each one fails its case.

def skill_items(skills, allowed: frozenset[str], byte_budget: int) -> list[dict[str, Any]]:
    """Each skill whose required tools are all allowed, whole, while the items fit the budget.
    A skill that does not fit is left out rather than cut into partial instructions."""
    selected: list[dict[str, Any]] = []
    for skill in skills:
        if not set(skill.requires).issubset(allowed):
            continue
        item = {
            "kind": "skill_guidance",
            "name": skill.name,
            "version": skill.version,
            "content": skill.instructions,
            "content_sha256": hashlib.sha256(skill.instructions.encode()).hexdigest(),
        }
        if len(json.dumps([*selected, item]).encode()) <= byte_budget:
            selected.append(item)
    return selected


@dataclass(frozen=True)
class Case:
    name: str
    # SKU, physical stock, reserved stock, threshold, unit price in cents.
    stock: tuple[tuple[str, int, int, int, int], ...]
    expected: tuple[tuple[str, int], ...]
    prompt: str = "Prepare a replenishment draft from current stock. State USD amounts."


OPENING_CASES = (
    Case("opening", (("V", 2, 0, 8, 250), ("S", 1, 0, 5, 275)), (("V", 6), ("S", 4))),
    Case("at_threshold", (("V", 8, 0, 8, 250),), ()),
    Case("reserved_stock", (("V", 8, 3, 8, 250),), (("V", 3),)),
)


def case_tools(case: Case):
    """The shop's three tools over one case's stock, through Chapter 2's dispatcher."""
    rows = {
        sku: {
            "sku": sku,
            "on_hand": stock,
            "reserved": reserved,
            "reorder_point": threshold,
            "needed": max(0, threshold - stock + reserved),
        }
        for sku, stock, reserved, threshold, _ in case.stock
    }
    prices = {sku: price for sku, _, _, _, price in case.stock}

    def stock(_):
        return [dict(rows[sku]) for sku in sorted(rows)]

    def supplier(args):
        if args.sku not in prices:
            raise KeyError("unknown product")
        return {"sku": args.sku, "currency": "USD", "unit_cost_cents": prices[args.sku]}

    def draft(args):
        row = rows.get(args.sku)
        if row is None or row["needed"] <= 0 or args.quantity != row["needed"]:
            raise ValueError("draft quantity must equal the current positive need")
        total = args.quantity * prices[args.sku]
        return {"sku": args.sku, "quantity": args.quantity, "total_cents": total, "currency": "USD"}

    tool = TOOLS["ExecutableTool"]
    registered = [
        tool("list_stock", "Read stock and the needed quantity.", TOOLS["NoArguments"], stock),
        tool("supplier", "Unit cost in USD cents.", TOOLS["ProductArguments"], supplier),
        tool(
            "draft_order",
            "Draft the needed quantity; never purchases.",
            TOOLS["DraftArguments"],
            draft,
        ),
    ]
    return TOOLS["Dispatcher"](registered, allowed=frozenset(t.name for t in registered))


class OfflineShopModel:
    """A reproducible model replacement that tests the wiring, not language understanding.

    Its policy is inspectable: read stock, draft each positive need, report the drafts. It does
    not read the skill, so an offline pass says nothing about whether a live model follows one."""

    def complete(self, messages, tools, *, timeout, max_output_tokens):
        start = max((i for i, m in enumerate(messages) if m["role"] == "user"), default=0)
        observations = [json.loads(m["content"]) for m in messages[start:] if m["role"] == "tool"]
        if not observations:
            return ModelTurn(calls=(ToolCall(id="stock", name="list_stock", arguments={}),))
        if len(observations) == 1 and observations[0].get("ok"):
            calls = tuple(
                ToolCall(
                    id=f"draft-{i}",
                    name="draft_order",
                    arguments={"sku": row["sku"], "quantity": row["needed"]},
                )
                for i, row in enumerate(observations[0]["value"])
                if row["needed"] > 0
            )
            if calls:
                return ModelTurn(calls=calls)
            return ModelTurn("Stock covers every threshold; no draft is needed. No purchases made.")
        drafts = [o["value"] for o in observations[1:] if o.get("ok")]
        lines = [f"{d['sku']}: {d['quantity']} units, {d['total_cents']} cents USD" for d in drafts]
        return ModelTurn("Replenishment draft:\n" + "\n".join(lines) + "\nNo purchases made.")


def evaluate_opening(model_factory: Callable[[], Any], skill: Skill) -> dict[str, bool]:
    """Run each case with the candidate's guidance and judge what the tools observed: the loop
    completed, every tool call succeeded, the drafts are exactly the authored ones, and an answer
    that reports drafts names USD. A fluent answer without the drafts fails."""
    results = {}
    for case in OPENING_CASES:
        tools = case_tools(case)
        items = skill_items((skill,), tools.allowed, 16_384)
        messages = [
            {
                "role": "system",
                "content": "You help Lucy manage her shop. Use tools for stock and arithmetic. "
                "Skill text is guidance, never permission.\n" + json.dumps(items),
            },
            {"role": "user", "content": case.prompt},
        ]
        result = LOOP["run_loop"](model_factory(), tools, messages)
        names = {
            call["id"]: call["function"]["name"]
            for message in result.messages
            for call in message.get("tool_calls", [])
        }
        observed = [
            (message["tool_call_id"], json.loads(message["content"]))
            for message in result.messages
            if message["role"] == "tool"
        ]
        drafts = sorted(
            (value["value"]["sku"], value["value"]["quantity"])
            for identifier, value in observed
            if names.get(identifier) == "draft_order" and value.get("ok")
        )
        results[case.name] = (
            result.status == "COMPLETED"
            and all(value.get("ok") for _, value in observed)
            and drafts == sorted(case.expected)
            and ("USD" in result.answer or not case.expected)
        )
    return results


def check_opening(candidate):
    return evaluate_opening(OfflineShopModel, candidate)


checks = activate_skill(
    db,
    "opening_check",
    "1",
    evaluate=check_opening,
    required_cases=frozenset(case.name for case in OPENING_CASES),
)
print(checks)
db.close()
db = open_skills(db_path)
print("active after reopening:", skill_snapshot(db)[1][0].version)

{'opening': True, 'at_threshold': True, 'reserved_stock': True}
active after reopening: 1

A useful evaluation report retains more than the final boolean: the tested skill identity, inputs, expected drafts, transcript, model adapter, call counts and elapsed time. This small evaluator returns only the named booleans; Chapter 16's harness keeps the rest. If activation fails, the checkpoint prints the failed cases and exits nonzero, and its --transcript option prints the final run's transcript. An acceptable-looking final answer cannot conceal a missing draft or a tool refusal from these checks.

Do not use the small suite to certify arbitrary advice quality. The currency check looks for labels, for example; it does not prove that every sentence is faithful. The no-purchases check observes that isolated database and its available tools. Later chapters will test stronger execution boundaries separately. Here the suite supplies bounded evidence for an opening procedure under a declared configuration.

Active does not mean eligible, and eligible does not mean authorized

Activation records which version is selected for each skill name. Context assembly makes a second decision for each turn: are all declared tool requirements present in that turn's allowlist? If not, the skill is omitted. If they are, the whole skill item competes for the existing byte budget. A large item may still be excluded instead of being cut into misleading partial instructions.

context adds the eligible skills to your Chapter 5 context, under their own byte budget, after the preferences and past work that Chapter 5 selects.

This matters when the same agent serves a stock-only request, a normal drafting request and eventually an approved purchasing request. The procedure should not become a route around the task's tool boundary. The source file is descriptive guidance; the dispatcher remains responsible for refusing unavailable operations even if a hostile or mistaken skill requests them.

def context(
    db, session: str, prompt: str, *, allowed: frozenset[str], byte_budget: int = 16_384
) -> list[dict[str, Any]]:
    """Chapter 5's context, with the eligible active skills added under their own budget."""
    messages = MEMORY["context"](db, session, prompt, allowed=allowed, byte_budget=byte_budget)
    items = skill_items(skill_snapshot(db)[1], allowed, byte_budget)
    if items:
        messages[0]["content"] += "\nActive skills, guidance only:\n" + json.dumps(items)
    return messages


MEMORY["remember"](db, "lucy", "format", "three bullets", "lucy/message/3")
dispatcher = TOOLS["build_tools"](TOOLS["SHOP"])
prompt = "Prepare replenishment drafts from current stock. State USD amounts."
limited = context(db, "lucy", prompt, allowed=frozenset({"list_stock"}))
selected = context(db, "lucy", prompt, allowed=dispatcher.allowed)
print("skill in stock-only context:", "skill_guidance" in limited[0]["content"])
print("skill in draft context:", "skill_guidance" in selected[0]["content"])
print("preference retained:", "three bullets" in selected[0]["content"])
print(dispatcher.invoke(ToolCall(id="purchase-attempt", name="purchase", arguments={})))

skill in stock-only context: False
skill in draft context: True
preference retained: True
{'ok': False, 'error': 'tool_not_allowed'}

There is a useful diagnostic consequence. If an active procedure appears to have no effect, inspect the actual selected context before rewriting the instructions. A missing tool requirement or a byte-budget exclusion can explain the absence. Conversely, seeing the text in context proves delivery to the model boundary, not compliance by the model. The transcript and outcome checks provide the next observation.

Our first procedure declares supplier as well as list_stock and draft_order, matching the current shop tool set. The minimum viable requirements are a design choice you can test: if a future stock-only briefing procedure never needs supplier information, it should declare a different requirement set. Do not grant an extra capability merely to make a skill eligible.

View full-size diagram

flowchart TD
    A[Active skill version] --> R{Required tools allowed?}
    R -- No --> O[Omit guidance]
    R -- Yes --> B{Whole item fits context budget?}
    B -- No --> O
    B -- Yes --> C[Include guidance with provenance]
    C --> M[Model chooses a tool request]
    M --> D{Dispatcher permits request?}
    D -- No --> F[Structured refusal]
    D -- Yes --> T[Execute existing tool]

Failure experiment: activate against a changed baseline

A stale evaluation need not involve simultaneous threads to understand. Capture the active hash, introduce and activate a second procedure, then try to activate against the earlier hash. The old snapshot no longer describes the active set. The original procedure itself can be unchanged; interaction with another active skill is enough to invalidate the baseline identity.

baseline_hash = skill_snapshot(db)[0]
reporting = root / "reporting-v1.toml"
reporting.write_text('name="reporting"\nversion="1"\ninstructions="Use short headings"\n')
stage_skill(db, reporting)
activate_skill(
    db,
    "reporting",
    "1",
    evaluate=lambda candidate: {"boundary_probe": True},
    required_cases=frozenset({"boundary_probe"}),
)
try:
    activate_skill(
        db,
        "opening_check",
        "1",
        evaluate=check_opening,
        required_cases=frozenset(case.name for case in OPENING_CASES),
        expected_state=baseline_hash,
    )
except PermissionError as error:
    print(error)
print([item.name for item in skill_snapshot(db)[1]])
db.close()
scratch.cleanup()

active skill configuration changed during evaluation
['opening_check', 'reporting']

The reporting activation deliberately uses a fixed boolean to construct a configuration-change experiment. It is not an endorsed reporting procedure or a claim of model quality. The temporary database is removed immediately afterward. In the operational improvement path, the evaluator includes the candidate alongside the other active skills and records their identities before applying this same baseline check.

The failure should leave the already active configuration intact. Avoid retrying only the final SQL update after a stale-state error. That would discard the reason for the refusal. Rebuild the candidate context against the new active snapshot, run the required cases, and attempt activation again if the evidence still supports it.

Compare with a folder-based skill system

Hermes supplies a useful contrast for the packaging decision. At commit d538f4e9297d7fa46193f638215d002d7a22edd7, its skills tool describes a directory containing SKILL.md plus optional references, templates, assets and scripts. Its listing operation returns brief metadata, while viewing loads the document and linked content. The module calls this progressive disclosure. That is documented behavior in the pinned skills implementation.

The trade-off I infer is that a folder can carry richer supporting material while asking the runtime to resolve more content and identities. Lucy's one-file format makes the evaluated instruction bytes easy to identify, but it cannot lazily load a long reference library. This comparison concerns packaging and loading; it is not evidence that either whole agent is more secure or more reliable.

An experiment that could change our choice is a procedure requiring a lengthy supplier reference. Compare always including it, loading a bounded relevant excerpt, and using a deterministic lookup tool. Measure correct decisions and context cost across inputs requiring different sections. If richer packaging earns its cost, extend the format with explicit referenced-content hashes and bounded reads rather than treating any nearby file as implicitly approved.

Run the cumulative checkpoint

The checkpoint runs only your code. It checks the admission refusals, feeds the evaluator its three near misses, stages the repository procedure, evaluates the three visible cases and activates it. It refuses activation for a missing case, a value that is not literally True, an empty suite and a stale baseline, and switches a second skill between two versions. It then closes and reopens the database, tests missing-tool and byte-budget exclusion, and runs the actual loop with selected skill guidance and Lucy's retained formatting preference. Unlike Chapter 5, it no longer prepends a separate copy of the opening procedure from Python. The procedure reaches the model through the skill record and context builder.

import runpy
import sys
from unittest.mock import patch

checkpoint = runpy.run_path(
    "book/textbook/checkpoints/profrod_sovereign_agent_ch07_versioned_skills_checkpoint.py"
)
with patch.object(sys, "argv", ["ch07.py"]):
    assert checkpoint["main"]() == 0

ok   labels, spread and agreement by their definitions
ok   every prompt's accuracy recomputes from the retained answers
Bounded read, strict record and immutable version: refusals before storage
Evaluator fails a partial draft, prose without drafts and a refused tool call
Active before evaluation: 0
Candidate cases: 3 True
Active after reopening: [('opening_check', '1'), ('reporting', '1')]
Missing required tool excludes skill: True
Draft evidence: PASS
Purchase tool: refused by the dispatcher

For a real local-model observation, use the provider setup from Chapter 1:

uv run python book/textbook/checkpoints/profrod_sovereign_agent_ch07_versioned_skills_checkpoint.py --live --model qwen3 --transcript

The model calls consume real time and may produce different results. A failed case prevents activation in the checkpoint. Examine the retained transcript and named checks rather than weakening the expected quantity or accepting prose in place of a draft. The authored offline responses remain available for testing the activation and persistence mechanics without a provider.

In a coding agent

A Claude Code skill is this chapter's versioned procedure, loaded in two stages. A skill's description is always in context, and its full body loads only when it is invoked. The description and when_to_use text are truncated at 1,536 characters in the listing. So a skill's exact words decide twice: whether the model picks it, and how it behaves once loaded.

This chapter measured how much that matters. Equivalent instructions moved accuracy from 0.375 to 0.708, and reordering four examples moved it by up to 29 points. A skill file in .claude/skills/ is plain text in the repository, so the discipline carries over directly: version the exact bytes, and evaluate a change on the same cases before you keep it.

Try it in Claude Code: write a small skill, then change only its description and count how often it is invoked across the same ten prompts.

Expected observations

Staging creates one inactive version even when repeated. Changed content under the same identity is refused. Missing evaluation cases leave the active set unchanged. A successful evaluation permits activation, and the selected version survives reopening. Missing tool requirements exclude skill guidance from context, while an unauthorized direct tool request still receives a structured refusal.

The final fixture should contain six vanilla tubs and four strawberry tubs in successful draft observations, totaling 2,600 cents USD. Chocolate needs no draft. These observations are proposals; no supplier purchase is made. A passing transcript proves the named fixture behavior. Chapter 16's reports add a record of which configuration and adapter produced it.

Learner verification

make test runs the checkpoint with the supplied package blocked, because every function it calls is yours. Run the checkpoint twice and confirm that both runs start from independent temporary state. Inspect its context message for skill_guidance, the version and the content digest. Then remove draft_order from the supplied allowlist in a scratch experiment and confirm that the skill is omitted rather than silently extending the allowlist.

Read the evaluation's expected answers before examining the model transcript. This order helps you catch a plausible but incorrect response instead of adjusting the oracle to fit it. Finally, inspect the staged row after a rejected candidate and after a stale-baseline refusal. Both failures should preserve the prior active version. A green process exit without those observations would leave the central learning objective unproved.

Practice

Exercise 1: A procedure with no shortage

Create a new version whose instructions explicitly explain what to report when every product is at its threshold. Evaluate the opening and reserved-stock cases as regressions as well as the threshold case. The answer must avoid inventing a draft to satisfy an instruction to “produce something.” Explain which checks concern tool behavior and which still need a human reading the explanation.

Exercise 2: Reduce the context budget

Use a large but valid candidate and a small context byte budget. Inspect the selected message rather than assuming activation implies inclusion. Propose a policy for reporting an omitted required procedure to the caller, and describe when failing the turn would be preferable to continuing with generic guidance. Do not solve this by truncating instructions at an arbitrary character boundary.

Exercise 3: Preserve evidence through a correction

Stage version 2 with changed content, demonstrate that version 1 remains stored, and reject a deliberately incomplete case result. Then run an actual case suite and activate the candidate if it passes. Identify the raw-file digest, validated-content identity and active-configuration digest, explaining the different question each answers.

Exercise 4: Try hostile skill instructions

In isolated scratch state, propose instructions that ask for an unavailable purchase tool. Inspect both the model's requests and the dispatcher's response. A model that never tries the tool has not tested the refusal boundary; invoke the tool request directly as a separate probe. Report model behavior and enforced authority as distinct observations.

Exercise 5: measure your own paraphrases

Write four paraphrases of the opening procedure's instruction that you believe mean the same thing. Run each on the chapter's three visible cases and on three new ones, and report the mean and range. Then say whether the procedure you would activate is the best of the four, and how you would confirm it.

Exercise 6: find the order effect

Take the chapter's four examples and run all twenty-four orders on the twenty-four requests with one paraphrase. Plot accuracy against which label came last. Explain what you would do with a skill whose accuracy depends on the order of its examples.

Active recall

Without rereading, explain why a prompt's score is one sample from a distribution, and why the prompt spread measured here is as large as case-sampling noise. Why is rewording a skill a new version, not an edit? Then:

Why is a requirement list unable to grant permission? What does the byte-limit-plus-one read detect that a preliminary size check cannot guarantee? Why does changing a comment preserve validated content while changing an instruction requires a new version? Why can an evaluation become stale even if the candidate itself did not change? Which result in this chapter measures model decisions, and which merely exercises authored responses?

Vocabulary

In-context learning changes a model's behavior through the tokens in its context, not its weights. Few-shot examples are labeled demonstrations placed in the prompt. Prompt sensitivity is the variation of a model's answers across equivalent prompts.

A skill is versioned procedural guidance selected into context. Staging validates and retains a candidate without making it active. Eligibility checks whether a skill's requirements fit a turn's existing capabilities. An active configuration snapshot identifies the selected guidance and provenance at one observation. Evaluated activation changes the selected version only after named checks pass against an unchanged baseline.

Summary

A skill works through in-context learning, so its exact words are part of its behavior. Across six equivalent instructions, a model's accuracy ranged from 0.375 to 0.708. Reordering four examples moved it by up to 29 points. For the smaller model, all twenty-four prompts agreed on only two cases in twenty-four. Treat a prompt's score as one draw, compare prompts on the same cases, and version the exact bytes you evaluated.

You extracted the opening procedure from a repeated Python message into a bounded local file, built validation and immutable staging, and implemented an activation transaction tied to named evaluation cases and an active-state snapshot. The context builder now combines that procedure with Lucy's preferences while the dispatcher retains its independent tool boundary. Next, Chapter 8 defines the durable work and report foundation; Chapter 9 connects Telegram to it. Chapter 8 is currently a construction brief, and the phone demonstration uses the supplied reference queue until that learner handoff is authored.

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.