Durable Agent Memory: Unit B Exercises

Student edition · 90 minutes of dedicated work · 2026-09-26

Runs on Google Colab as it ships today (Python 3.12), or on any local Python 3.12+ kernel. It needs nothing beyond Python's standard library, and it makes no network call.

This is the second of Chapter 5's two practical units. Unit A made Lucy's preferences durable. This unit decides which stored notes reach the model.

A colleague has written the agent's memory search. It counts the words a note shares with the request and sends the top three notes. It "works on every question I tried". You will build BM25, the standard lexical ranking function, and measure both rankers against labeled questions. Then you will write the metric that says how early the right note appears.

By the end you should be able to:

Explain recall@k and reciprocal rank, and why recall is the measure that matters to an agent.
Derive BM25 from three ideas: rare terms matter more, repetition saturates, and long records are discounted.
Evaluate rankers against labeled questions, with a negative control, and separate direct questions from paraphrases.
Say what no lexical ranker can do, and what would.

Minutes	Dedicated work	Saved evidence
0–10	Predict which questions the colleague's search misses	Written prediction
10–30	Ranking metrics and the three ideas behind BM25	Worked outputs and handoff
30–60	Construct `bm25_scores` and pass the visible cases	Learner code and grade table
60–70	Evaluate three rankers on Lucy's labeled questions	Recall table and negative control
70–85	Changed-constraint task: mean reciprocal rank	Transfer results
85–90	Explain the result and save evidence	Retained submission

These times are planning estimates, not measured completion times. Run All only checks that the notebook executes; the unfinished student functions deliberately report NEEDS_WORK. Keep your first attempt before you open the answers.

Run the self-contained setup

The unit needs only Python's standard library. The collapsed cell supplies:

NOTES: forty of Lucy's shop notes, the agent's memory;
QUESTIONS: sixteen questions, each labeled with the one note that answers it (and the expected answer, unused here). The last four are paraphrases that share few words with their note;
terms, the tokenizer, and the colleague's overlap_scores, plus recent_scores, which ranks the newest notes first.

These are the notes and questions of the chapter's experiment. Run setup on every fresh kernel. Your saved work lives in practical-work/ch05-b.

Supplied setup: Lucy's notes, labeled questions and two simple rankers

The complete supplied setup cell is in the downloadable notebook. Run it there before attempting the cells below. This web edition omits only that compressed setup payload.

Commit to a prediction before the examples

The colleague's ranker counts shared words. Read the sixteen questions and their notes in QUESTIONS and NOTES, then write down:

which questions you expect it to miss in its top three;
whether you expect BM25 to fix all of them.

python

prediction_notes = {
    "prediction": "List the questions word overlap will miss, and whether BM25 fixes them.",
    "reason": "Name the rule behind that prediction.",
    "falsifier": "Name an observation that would prove the explanation wrong.",
    "revision": "After execution, explain what changed in your understanding.",
}

Retrieval is ranking, and ranking has metrics

A retriever orders the notes for a request. The notes that answer it are relevant; here, each question's relevant note is labeled in advance. Two measures matter most to an agent:

Recall@k: the share of relevant notes in the top $k$ . A relevant note outside the top $k$ never reaches the model, so a missed recall is a question the agent cannot answer from memory.
Reciprocal rank: $1/r$ for the first relevant note at position $r$ , or 0 if none is returned. It rewards putting the answer first.

Predict both for the ranking in the cell before you run it.

python

intro_ranked, intro_relevant = [4, 1, 7, 2], {1}
intro_recall = sum(i in intro_relevant for i in intro_ranked[:3]) / len(intro_relevant)
intro_first = next(p for p, i in enumerate(intro_ranked, start=1) if i in intro_relevant)
intro_rr = 1 / intro_first
print("recall@3", intro_recall, "reciprocal rank", intro_rr)

Three ideas behind BM25

Counting shared words treats "the" like "Hartwell". Three refinements fix that.

Rare terms matter more. A term in $n_t$ of $N$ notes is weighted by its inverse document frequency, $\mathrm{idf}(t) = \ln\bigl(1 + \frac{N - n_t + 0.5}{n_t + 0.5}\bigr)$ . A term in every note is worth almost nothing.
Repetition saturates. A term appearing $f$ times contributes $\frac{f(k_1 + 1)}{f + k_1}$ , which rises with $f$ but never exceeds $k_1 + 1$ .
Long notes are discounted. $k_1$ in the denominator becomes $k_1\bigl(1 - b + b\,\frac{|d|}{\overline{|d|}}\bigr)$ , where $|d|$ is the note's length in terms and $\overline{|d|}$ the average.

\mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{idf}(t)\,\frac{f_{t,d}\,(k_1 + 1)}{f_{t,d} + k_1\left(1 - b + b\,\frac{|d|}{\overline{|d|}}\right)}, \qquad k_1 = 1.5,\ b = 0.75.

Each distinct query term counts once; a term absent from a note contributes nothing.

Choose an explicit starting point

This unit starts from Unit A's work. To use your own, replace None with the path to your practical-work/ch05-a/ch05-unit-a-handoff-v1.json. Leave it as None to start from the supplied reference. Your submission records which you chose.

python

LEARNER_HANDOFF = None

python

import hashlib
import shutil

COURSE_INPUT = COURSE_WORK / "ch05-unit-a-handoff-v1.json"
if LEARNER_HANDOFF is not None:
    learner_input = Path(LEARNER_HANDOFF).expanduser().resolve()
    if not learner_input.is_file():
        raise FileNotFoundError("The selected learner handoff does not exist")
    if learner_input != COURSE_INPUT.resolve():
        shutil.copy2(learner_input, COURSE_INPUT)
    HANDOFF_ORIGIN = "LEARNER_SELECTED"
else:
    reference_source = (
        "def remember(db, session, name, value, source):\n    ...  # Supplied reference.\n"
    )
    reference_handoff = {
        "schema": 1,
        "status": "VISIBLE_PASSED",
        "implementation": reference_source,
        "implementation_sha256": hashlib.sha256(reference_source.encode()).hexdigest(),
    }
    COURSE_INPUT.write_text(json.dumps(reference_handoff, indent=2, sort_keys=True) + "\n")
    HANDOFF_ORIGIN = "SUPPLIED_REFERENCE"
print("Starting evidence:", HANDOFF_ORIGIN)

python

handoff = json.loads(COURSE_INPUT.read_text(encoding="utf-8"))
source_matches = isinstance(handoff.get("implementation"), str) and hashlib.sha256(
    handoff["implementation"].encode()
).hexdigest() == handoff.get("implementation_sha256")
handoff_status = (
    "VERIFIED" if source_matches and handoff.get("status") == "VISIBLE_PASSED" else "INVALID"
)
print("UNIT_A_HANDOFF", handoff_status)

Unit A decided what Lucy's memory keeps. This unit decides which of those records a given request sees. A record that is stored but never retrieved is, for that request, not remembered at all.

Main practical: construct, connect and challenge

1. Construct `bm25_scores`

bm25_scores(query, documents, k1=1.5, b=0.75) must return one score per document, by the formula above, using terms to split text. It must also:

return an empty list for no documents;
raise ValueError when k1 is negative or b lies outside $[0, 1]$ .

The starter below is the colleague's overlap count. Run the visible cases, then repair it.

python

def bm25_scores(query, documents, k1=1.5, b=0.75):
    """Score each document for the query; higher is more relevant."""
    wanted = set(terms(query))
    return [float(len(wanted & set(terms(d)))) for d in documents]

Hint 1: the pieces

Tokenize every document once. Count, for each term, how many documents contain it: that is $n_t$ . The average length is the mean number of terms per document.

Hint 2: the first visible case

With three documents of equal length and a term in only one of them, the length factor is 1 and the saturation term is $\frac{1 \cdot 2.5}{1 + 1.5} = 1$ . The score is just $\mathrm{idf} = \ln(1 + 2.5/1.5)$ .

python

import copy


def outcome(call):
    try:
        return [round(x, 6) for x in call()]
    except Exception as error:
        return type(error).__name__


VISIBLE_CASES = [
    (
        "a term in one of three equal notes scores its idf",
        lambda: bm25_scores("a", ["a b", "b c", "c d"]),
        [round(math.log(1 + 2.5 / 1.5), 6), 0.0, 0.0],
    ),
    (
        "eight repeats score less than eight times one",
        lambda: bm25_scores("a", ["a a a a a a a a", "a b c d e f g h"]),
        [round(math.log(1.2) * 8 * 2.5 / 9.5, 6), round(math.log(1.2), 6)],
    ),
    (
        "the rare shared term wins",
        lambda: ranking(
            bm25_scores("the note hartwell", ["hartwell", "the note", "the other note"])
        )[:1],
        [0],
    ),
    (
        "a shorter note with the same term ranks higher",
        lambda: ranking(bm25_scores("mint", ["mint tea and cake and coffee", "mint"]))[:1],
        [1],
    ),
    ("no documents", lambda: bm25_scores("a", []), []),
    ("a negative k1 is refused", lambda: bm25_scores("a", ["a"], k1=-1), "ValueError"),
    ("b above one is refused", lambda: bm25_scores("a", ["a"], b=1.5), "ValueError"),
]


def grade(cases):
    rows = []
    for label, call, expected in cases:
        observed = outcome(call)
        rows.append(
            {
                "case": label,
                "expected": expected,
                "observed": observed,
                "status": "PASS" if observed == expected else "FAIL",
            }
        )
    return rows


visible_results = grade(VISIBLE_CASES)
VISIBLE_PASSED = all(row["status"] == "PASS" for row in visible_results)
for visible_row in visible_results:
    print(visible_row["status"], visible_row["case"], "->", visible_row["observed"])
print("VISIBLE_CONTRACT", "PASSED" if VISIBLE_PASSED else "NEEDS_WORK")

2. Connect: evaluate three rankers on Lucy's labeled questions

The cell below ranks the forty notes for every question with your bm25_scores, with the colleague's overlap count and with "newest first". It reports recall@3 on the twelve direct questions and on the four paraphrases.

Your BM25 must find every direct question's note in its top three. "Newest first" is the negative control: an evaluation that cannot tell it apart from a real ranker proves nothing.

Predict each ranker's recall on the paraphrases.

python

connected = None
PARAPHRASES = set(range(12, 16))


def recall_at_3(scorer, keep):
    hits = [
        relevant & set(ranking(scorer(question, NOTES))[:3]) != set()
        for index, (question, relevant, _) in enumerate(QUESTIONS)
        if keep(index)
    ]
    return round(sum(hits) / len(hits), 3)


if VISIBLE_PASSED:
    table = {}
    for name, scorer in (
        ("bm25", bm25_scores),
        ("word_overlap", overlap_scores),
        ("newest_first", recent_scores),
    ):
        table[name] = {
            "direct": recall_at_3(scorer, lambda i: i not in PARAPHRASES),
            "paraphrases": recall_at_3(scorer, lambda i: i in PARAPHRASES),
        }
    finds_direct = table["bm25"]["direct"] == 1.0
    control_rejected = table["newest_first"]["direct"] < 0.2
    missed = [
        question
        for index, (question, relevant, _) in enumerate(QUESTIONS)
        if not relevant & set(ranking(bm25_scores(question, NOTES))[:3])
    ]
    connected = {
        "table": table,
        "finds_direct": finds_direct,
        "negative_control_rejected": control_rejected,
        "bm25_missed": missed,
    }
    for name, row in table.items():
        print(
            f"{name:13} recall@3 direct {row['direct']:.3f}  paraphrases {row['paraphrases']:.3f}"
        )
    print("BM25 finds every direct question's note:", finds_direct)
    print("negative control rejected:", control_rejected)
    print("BM25 misses:", missed)
else:
    print("CONNECTION_NOT_READY — repair bm25_scores, then run again.")

Both word-based rankers are perfect on direct questions, and both fail paraphrases: "At what hour should the supplier's van turn up?" shares no useful word with "Lucy asked for deliveries in the afternoon". No lexical ranker can find a note that says the same thing in different words. That needs an embedding model, which maps meaning rather than words to vectors. It is also why the chapter measured whether sending the whole memory would be better: for forty notes, it was.

Exit ticket

Answer three questions:

Why is recall@k, not precision@k, the measure that decides what an agent can answer from memory?
Which of BM25's three ideas moved the rare-term case, and which moved the repeated-term case?
Your colleague "tried every question" and saw good answers. Name two ways a good-looking answer can hide a retrieval miss.

python

exercise_report = {
    "unit": "ch05-b",
    "attempted": 1,
    "completed": int(VISIBLE_PASSED),
    "failed": int(not VISIBLE_PASSED),
    "skipped": 0,
    "connection": "PASSED" if connected else "NOT_READY",
    "handoff": handoff_status,
}
print("EXERCISE_REPORT=" + json.dumps(exercise_report, sort_keys=True))

Changed-constraint construction: mean reciprocal rank

Allow fifteen minutes: three to predict, eight to implement and check, and four for a case of your own.

Recall@3 says whether the note arrived. Reciprocal rank says how early. Averaged over many requests, it is the mean reciprocal rank (MRR).

Implement mean_reciprocal_rank(rankings, relevant_sets):

rankings is a list of ranked index lists, one per request; relevant_sets the matching sets of relevant indices;
each request scores $1/r$ for the first relevant index at position $r$ (from 1), or 0 if none appears;
return the mean over requests;
raise ValueError when the lists are empty or of different lengths, or when any relevant set is empty;
do not change the inputs.

Hint: positions start at one <code>enumerate(ranked, start=1)</code> gives each index with its position.

python

def mean_reciprocal_rank(rankings, relevant_sets):
    raise NotImplementedError("Score 1 / position of the first relevant index, then average")

python

TRANSFER_CASES = [
    ("first at position two", [[[3, 1, 2]], [{1}]], 0.5),
    ("first and second", [[[0], [1, 0]], [{0}, {0}]], 0.75),
    ("never returned", [[[2, 3]], [{9}]], 0.0),
    ("the first of several relevant counts", [[[5, 4, 3]], [{4, 3}]], 0.5),
    ("no requests are refused", [[], []], {"raises": "ValueError"}),
    ("mismatched lists are refused", [[[1]], [{1}, {2}]], {"raises": "ValueError"}),
    ("an empty relevant set is refused", [[[1]], [set()]], {"raises": "ValueError"}),
]


def run_transfer(candidate, cases):
    observations = []
    for label, arguments, expected in cases:
        supplied = copy.deepcopy(arguments)
        try:
            actual = round(candidate(*supplied), 6) + 0.0
        except NotImplementedError:
            actual = {"unfinished": True}
        except Exception as error:
            actual = {"raises": type(error).__name__}
        passed = actual == expected and supplied == arguments
        observations.append(
            {"case": label, "expected": expected, "observed": actual, "passed": passed}
        )
        print("PASS" if passed else "NEEDS_WORK", label, "expected", expected, "observed", actual)
    return observations


transfer_observations = run_transfer(mean_reciprocal_rank, TRANSFER_CASES)
TRANSFER_PASSED = all(row["passed"] for row in transfer_observations)
print("TRANSFER_STATUS", "PASS" if TRANSFER_PASSED else "NEEDS_WORK")

Design a counterexample and retrieve the mechanism

Add a case with an expected value you worked out by hand, and rerun the driver. Then compute MRR over the sixteen questions for BM25 and for word overlap, using your function and ranking. The chapter measured 0.791 and 0.804: overlap puts the answer first slightly more often, although BM25 has the better recall. Explain how both can be true, and which you would optimize for an agent that sends three notes.

Save your evidence and explain the result

Fill in the prediction notes and your explanation before saving. Include:

the exact observed value, and the input that caused it;
your code's invocation point;
one failed hypothesis;
the strongest claim the evidence still cannot support.

Sixteen labeled questions over forty notes is a small evaluation. It shows how lexical ranking fails, not how often it would fail on Lucy's real requests.

python

explanation_notes = {
    "causal_trace": "Explain the input, learner invocation and observed result.",
    "failed_hypothesis": "Describe a prediction the evidence changed.",
    "remaining_limit": "Name the guarantee not established by this experiment.",
}
course_submission = {
    "unit": "ch05-b",
    "planned_minutes": 90,
    "starting_evidence": globals().get("HANDOFF_ORIGIN", "INDEPENDENT_UNIT_A"),
    "prediction": prediction_notes,
    "explanation": explanation_notes,
    "core_report": exercise_report,
    "transfer": transfer_observations,
    "explanation_review": "HUMAN_REVIEW_REQUIRED",
}
submission_path = COURSE_WORK / "ch05-b-submission-v1.json"
submission_path.write_text(
    json.dumps(course_submission, indent=2, sort_keys=True), encoding="utf-8"
)
print("Saved evidence:", submission_path)
print(
    "COURSE_REPORT="
    + json.dumps(
        {
            "unit": "ch05-b",
            "transfer_passed": TRANSFER_PASSED,
            "starting_evidence": course_submission["starting_evidence"],
            "edition": "student",
        },
        sort_keys=True,
    )
)

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.