Embeddings and Vector Search: Unit B Exercises

Student edition · 90 minutes of dedicated work · 2026-09-27

Runs on Google Colab as it ships today (Python 3.12), or on any local Python 3.12+ kernel. It needs nothing beyond Python's standard library, and no model or key.

This is the second of Chapter 6's two practical units. Unit A built a vector store from vectors you trained yourself. This unit asks how good a retriever is, measures three of them on labeled questions, and repairs what the measurement shows.

The three retrievers are:

BM25, Chapter 5's word matcher;
your vectors from Unit A, trained on twelve notes;
all-minilm, a real sentence encoder with 384 numbers per vector. Its vectors for Lucy's notes and questions were computed ahead of time and ship inside this notebook, so nothing is downloaded. It is the same model as Chroma's default embedding function, all-MiniLM-L6-v2.

By the end you should be able to:

Explain recall@k and reciprocal rank, and compute them by hand for one question.
Implement evaluate, which scores a retriever's rankings against labeled questions.
Measure three retrievers on direct and paraphrased questions, and explain the difference.
Weight a fusion of two rankings, and check whether it beats its best part.

Minutes	Dedicated work	Saved evidence
0–10	Predict which retriever finds paraphrases	Written prediction
10–30	Labeled questions, the metrics, and your starting evidence	Worked outputs and handoff
30–55	Construct `evaluate` and pass the visible cases	Learner code and grade table
55–70	Measure three retrievers on Lucy's questions	Independent observation
70–85	Changed-constraint task: weighted fusion	Transfer results
85–90	Explain the result and save evidence	Retained submission

These times are planning estimates, not measured completion times. Run All only checks that the notebook executes; the unfinished student functions deliberately report NEEDS_WORK. Keep your first attempt before you open the answers.

Run the self-contained setup

The collapsed cell below creates your work folder and defines the supplied parts of the unit:

Lucy's twelve notes and twelve labeled questions: six asked in the notes' own words, and six paraphrases that share none of their words.
BM25, as in Chapter 5, returning only notes that share at least one word with the question.
Vector arithmetic, note_vector from Unit A, and a supplied search that behaves like the one you built.
all-minilm's vectors for every note and question, computed with the real model through Ollama on September 27, 2026 and rounded to five decimals.

Run setup on every fresh kernel. Your saved work lives in practical-work/ch06-b.

Supplied setup, questions, BM25 and the real model's vectors

The complete supplied setup cell is in the downloadable notebook. Run it there before attempting the cells below. This web edition omits only that compressed setup payload.

Commit to a prediction before the examples

Six of the questions paraphrase a note without using any of its words: "a treat without milk" for "mango sorbet is dairy-free and vegan". Before you run anything below, write down:

which of the three retrievers you expect to find those notes;
why your Unit A vectors might not even produce a ranking for them.

python

prediction_notes = {
    "prediction": "Write which retriever finds the paraphrased notes, and why.",
    "reason": "Name the rule behind that prediction.",
    "falsifier": "Name an observation that would prove the explanation wrong.",
    "revision": "After execution, explain what changed in your understanding.",
}

Two metrics, by hand

A ranking lists note ids, best first. A labeled question names the notes that answer it. Two numbers describe how good a ranking is:

recall@k: the share of the relevant notes that appear in the first $k$ ;
reciprocal rank: $1/r$ , where $r$ is the position of the first relevant note, or 0 if none appears.

For "what goes in a waffle cone", the relevant notes are n5 and n6. Predict both numbers for the ranking below with $k = 2$ , then run it.

python

ranking = ["n6", "n8", "n5", "n7"]
relevant = ["n5", "n6"]
print("recall@2:", len(set(ranking[:2]) & set(relevant)) / len(relevant))
first = next(position for position, note in enumerate(ranking, start=1) if note in relevant)
print("reciprocal rank:", 1 / first)

What each retriever does with one paraphrase

The cell below asks "a dessert children like", whose answer is n8, "kids want a chocolate cone". Predict each retriever's first answer.

python

question = "a dessert children like"
print("BM25:", bm25_ranking(question)[:3])
print("all-minilm:", minilm_ranking(question)[:3])

Choose an explicit starting point

This unit measures the index you built in Unit A. To use your own, replace None with the path to your practical-work/ch06-a/ch06-unit-a-handoff-v1.json. Leave it as None to start from the supplied reference, which is the index the worked Unit A produces. Your submission records which you chose.

python

LEARNER_HANDOFF = None

python

import shutil

COURSE_INPUT = COURSE_WORK / "ch06-unit-a-handoff-v1.json"
if LEARNER_HANDOFF is not None:
    learner_input = Path(LEARNER_HANDOFF).expanduser().resolve()
    if not learner_input.is_file():
        raise FileNotFoundError("The selected learner handoff does not exist")
    if learner_input != COURSE_INPUT.resolve():
        shutil.copy2(learner_input, COURSE_INPUT)
    HANDOFF_ORIGIN = "LEARNER_SELECTED"
else:
    reference_encoded = (
        "c-pO4%Wfh`7Tx<RLac3c-;df(_pV1Xi`k88kp<~0)HZ2lVpo@1_3xtqTx3vc*)+q#NE!HW&pr2Xf<JFcS!}lJ{*T"
        "3n^52_MO8ImDx0J=|c^v!gZt<b)#^?U__3>uC_)r#GT>N<QIIR1J`fyr}-EQ^ej@GBcUH?agSUh&y&qJAp@l(Gmo"
        "2jh3&G_x^e(d|wZP(?c|J-d)|Mk+ZcEk81ruw>rWpc1Z1}Oxx=<WQFMU54OLs`lBc=mDs{{8k6Zge5Hq1%<0Zo7G"
        "S=>9gffP_fU3=)Z!8m5T32L`Go<2Bs)LT>x$%j$XB4Uhdem2UdI+z;bX$+s2ra3x9yprkxg%+kp+6vdrLub?Ft(w"
        "b@XGq;(ZZ|lY&qzP=64A$xv%<dejOw8(b1wp-3w_`m{t6_L5n{DaJ*Y5uQp)adp+rLfA1Dx1XRZ;Oo3zn#Afl0{D"
        "8SZ%nE4z@@>dUYi9=cusw|RkSl?vTNJR4rAz8@^%0AS`<@bU|JO}nxC`l%nsZ*yYP^XG1SCofhbM-)uM`S_?+2x="
        "B4i2{;9SMZ7pdHudwPvvX3-IcDKw0jpL2T%gD6y+qJGa{i#2S*dvV0v#~-FFYuPn7aPO1m$8nVz4X9ya}0*8Rigr"
        "H+o>^QWl{+wwSU*WI_b$f2cCB)f!bDEe(HrYZygd1AQ96?!3D!*1%g>#5X7o9)Z6S@o6S{kVAtLn6sEHP8rhwRSM"
        "33u~&n6B_tc3@<ezPu;iDuzcmV*_Ktm-A!*(Lkb~Qb`qmFr=|qSs%n^F*Q{#1*oTbkek_l}*dOZgejFYfu5Z?5Fs"
        "Ei}DjWS&mJFPTp&<!)4Ke<|_T2j_5`#D*&^-v~oEecx2&$v=+2Fq^<zxN$UxvDvcHQoIswe;W*FXRL+pqum^_RoJ"
        "^LBGwTVQtwFR}PKjO+T@L#`Hwk~_tE2g^WWb9Z54iytD~@e<t8GRZV3fDe81@%D)HI$!5#v=l)|tAmDY=D=WS$&7"
        "?9ni`?ng4Ge-;3&Hj1PURho>mMj4Z(y}y=Ir#TA){VHAllGN@kcisad9;NWyY&^1&>s&u}Ae@J`09O<IPM)a{0eD"
        "L4ltB@lP!OsE^B#XDA0izWmr5Xl^&X#Atc?`RoV$9MJV9unS)aID1!Ol!i-%`=^aafXG}hahBxrcfK%aW1u3T#Y8"
        "X><tQn{gB}fmuaG&NC~Ni(B`df;<Lg?HRJFqL<IOr&>C#@e)S6ygBqayMXnVSk}?=GSpp^Ixfctrek%e|Q?P|PEX"
        "%XZb11Mlm|}3wlEqFTYy<)depVMlw`+q^5nyl-N=^iwR~YV5oDw<IS_St*w2jF)Bm$(~(HLe>yrgtSA_#HBXpx@I"
        "xw)&TyQyI$A<b}*1k4>o(4AVGy(4V33qZDN&IwIz){C5sU>sT76ePJtstf%rCf0D$sB_&jb(U`-Czm=RYT0Isbh!"
        "StEy(Q1R$ZF}w;4B+Krm!?3v6f2$^5noCW%2cdKPImFHlM)Hbu+&WdLV*)i$5iI++qOkngW#CnTxm?}}dG1pN%?x"
        "Hiw)nK`81i_8G9$rzKGrw6H|M;lgMS7*&!D_yBBBvW9XAt?kY8G_V4_h^t_@1{+*7)1yGg}?}nPaFX~NXWz>(<Hf"
        "V=)><5?Q|w{vIt@k(YB%4RaGQ1u>NAvKpk#kb1YeclccaZDG1L}Z&ufFq<HnYM?+C}?)RJTH~#}}gc2M"
    )
    COURSE_INPUT.write_bytes(zlib.decompress(base64.b85decode(reference_encoded)))
    HANDOFF_ORIGIN = "SUPPLIED_REFERENCE"
print("Starting evidence:", HANDOFF_ORIGIN)

python

handoff = json.loads(COURSE_INPUT.read_text(encoding="utf-8"))
handoff_status = (
    "VERIFIED"
    if handoff.get("status") == "COMPLETED" and handoff.get("unit") == "ch06-a"
    else "INVALID"
)
own_words, own_index, own_model = handoff["words"], handoff["index"], handoff["model"]


def own_ranking(question):
    """Your Unit A index: no ranking at all when the question has no word the vectors know."""
    try:
        vector = note_vector(question, own_words)
    except ValueError:
        return []
    return search(own_index, {"model": own_model, "vector": vector}, len(own_index))


print("UNIT_A_HANDOFF", handoff_status, "|", len(own_words), "words,", len(own_index), "index rows")

2. Construct `evaluate`

evaluate(rankings, relevant, k) receives a list of rankings (one per question, each a list of note ids, best first), a list of the same length holding each question's relevant note ids, and an integer k. It must return a dictionary with:

"recall_at_k": the mean over questions of the share of relevant ids in the ranking's first k;
"reciprocal_rank": the mean over questions of $1/r$ for the first relevant id at position $r$ , or 0 when none appears anywhere in the ranking;

both rounded to three decimals. It must raise ValueError when the two lists have different lengths, when there are no questions, or when k is less than 1. An empty ranking scores 0 on both. Change neither input.

The starter below counts a question as found when anything relevant is in the first k, and takes the reciprocal rank only within the first k. Run the visible cases to see where it fails, then repair it.

python

def evaluate(rankings, relevant, k):
    """Mean recall@k and mean reciprocal rank over the questions."""
    found, reciprocal = 0.0, 0.0
    for ranking, answers in zip(rankings, relevant, strict=False):
        top = ranking[:k]
        found += 1.0 if set(top) & set(answers) else 0.0
        for position, note in enumerate(top, start=1):
            if note in answers:
                reciprocal += 1 / position
                break
    return {
        "recall_at_k": round(found / len(rankings), 3),
        "reciprocal_rank": round(reciprocal / len(rankings), 3),
    }

Hint 1: recall is a share, not a hit

A question with two relevant notes and one of them in the top $k$ has recall 0.5, not 1.

Hint 2: reciprocal rank looks at the whole ranking

A first relevant note at position 4 is worth $1/4$ even when $k = 2$ . The two metrics answer different questions: whether the answer is near the top, and how far down the first one is.

Hint 3: refuse bad input

zip stops at the shorter list without complaint, which would quietly drop questions. Check the lengths first, or pass strict=True.

python

import copy

VISIBLE_CASES = [
    (
        "a perfect ranking",
        [[["n1", "n2"]], [["n1"]], 1],
        {"recall_at_k": 1.0, "reciprocal_rank": 1.0},
    ),
    (
        "half the relevant notes",
        [[["n5", "n8"]], [["n5", "n6"]], 2],
        {"recall_at_k": 0.5, "reciprocal_rank": 1.0},
    ),
    (
        "found below k",
        [[["n2", "n3", "n4", "n1"]], [["n1"]], 2],
        {"recall_at_k": 0.0, "reciprocal_rank": 0.25},
    ),
    (
        "an empty ranking",
        [[[], ["n1"]], [["n1"], ["n1"]], 1],
        {"recall_at_k": 0.5, "reciprocal_rank": 0.5},
    ),
    (
        "means over questions",
        [[["n1"], ["n2", "n1"], ["n3"]], [["n1"], ["n1"], ["n1"]], 2],
        {"recall_at_k": 0.667, "reciprocal_rank": 0.5},
    ),
    ("different lengths are refused", [[["n1"]], [["n1"], ["n2"]], 1], "ValueError"),
    ("k below one is refused", [[["n1"]], [["n1"]], 0], "ValueError"),
]


def grade_evaluate(candidate, cases):
    rows = []
    for label, arguments, expected in cases:
        supplied = copy.deepcopy(arguments)
        try:
            observed = candidate(*supplied)
        except Exception as error:
            observed = type(error).__name__
        passed = observed == expected and supplied == arguments
        rows.append(
            {
                "case": label,
                "expected": expected,
                "observed": observed,
                "status": "PASS" if passed else "FAIL",
            }
        )
    return rows


visible_results = grade_evaluate(evaluate, VISIBLE_CASES)
VISIBLE_PASSED = all(r["status"] == "PASS" for r in visible_results)
for visible_row in visible_results:
    print(visible_row["status"], visible_row["case"], "->", visible_row["observed"])
print("VISIBLE_CONTRACT", "PASSED" if VISIBLE_PASSED else "NEEDS_WORK")

3. Measure three retrievers on Lucy's questions

Once the visible cases pass, the cell below ranks the twelve notes for every question with BM25, with your Unit A index and with all-minilm, and scores each retriever with your evaluate, separately for direct questions and paraphrases, at $k = 2$ .

Before running it, predict the order of the three retrievers on paraphrases.

python

connected = None
if VISIBLE_PASSED:
    rankers = {"bm25": bm25_ranking, "yours": own_ranking, "all-minilm": minilm_ranking}
    table = {}
    for name, ranker in rankers.items():
        for kind in ("direct", "paraphrase"):
            chosen = [(q, answers) for q, answers, k in QUESTIONS if k == kind]
            scores = evaluate([ranker(q) for q, _ in chosen], [answers for _, answers in chosen], 2)
            table[f"{name} {kind}"] = scores
            recall, rank = scores["recall_at_k"], scores["reciprocal_rank"]
            print(f"{name:10} {kind:10} recall@2 {recall:.3f}  reciprocal rank {rank:.3f}")
    unranked = [q for q, _, kind in QUESTIONS if kind == "paraphrase" and not own_ranking(q)]
    print("paraphrases your vectors could not rank:", len(unranked), "of 6")
    connected = {"table": table, "unranked": unranked}
    paraphrase = {name: table[f"{name} paraphrase"]["recall_at_k"] for name in rankers}
    assert paraphrase["all-minilm"] > paraphrase["bm25"] > paraphrase["yours"] == 0.0

else:
    print("CONNECTION_NOT_READY — repair evaluate, then run again.")

Your vectors know only the twenty-four words of Lucy's notes, so a question in other words gives them nothing to average. BM25 ranks only notes that share a word; on paraphrases that is usually a small word such as "a" or "the". Its right answer for "a dessert children like" came from the word "a" in a short note, which is luck, not understanding. all-minilm learned from about a billion sentence pairs that "children" and "kids" mean the same.

4. Save the measurement

The course report below keeps the table. Chapter 6 measures the same three kinds of retriever on Chapter 5's forty notes, with a larger set of questions.

python

exercise_report = {
    "unit": "ch06-b",
    "attempted": 1,
    "completed": int(VISIBLE_PASSED),
    "failed": int(not VISIBLE_PASSED),
    "skipped": 0,
    "connection": "PASSED" if connected else "NOT_READY",
    "handoff": handoff_status,
}
print("EXERCISE_REPORT=" + json.dumps(exercise_report, sort_keys=True))

Changed-constraint construction: weighted fusion

Allow fifteen minutes: three to predict, eight to implement and trace, and four to search for a weight.

Unit A fused two rankings with equal weight. When one ranker is weak on a kind of question, equal weight lets it pull good answers down. Give each ranking a weight: an id scores the sum over rankings of $w/(60 + \text{rank})$ .

transfer_weighted_fuse(rankings, weights, k) receives a list of rankings, a list of the same length of non-negative weights, and k. Return the k best ids by weighted score, best first, ties by id. Raise ValueError if the lengths differ or a weight is negative. An id missing from a ranking gets nothing from it. Change neither input.

Write your expected values before you run the table.

python

def transfer_weighted_fuse(rankings, weights, k):
    raise NotImplementedError("Sum w / (60 + rank) for each id, then take the k best")

python

TRANSFER_CASES = [
    ("equal weights are ordinary fusion", [[["n1", "n2"], ["n2", "n1"]], [1, 1], 2], ["n1", "n2"]),
    ("a heavier ranking wins", [[["n1", "n2"], ["n2", "n1"]], [1, 2], 2], ["n2", "n1"]),
    ("a zero weight silences a ranking", [[["n1"], ["n2"]], [0, 1], 2], ["n2", "n1"]),
    ("an empty ranking adds nothing", [[[], ["n3", "n1"]], [5, 1], 1], ["n3"]),
    ("a negative weight is refused", [[["n1"]], [-1], 1], {"raises": "ValueError"}),
]


def run_transfer(candidate, cases):
    observations = []
    for label, arguments, expected in cases:
        supplied = copy.deepcopy(arguments)
        try:
            actual = candidate(*supplied)
        except NotImplementedError:
            actual = {"unfinished": True}
        except Exception as error:
            actual = {"raises": type(error).__name__}
        passed = actual == expected and supplied == arguments
        observations.append(
            {"case": label, "expected": expected, "observed": actual, "passed": passed}
        )
        print("PASS" if passed else "NEEDS_WORK", label, "expected", expected, "observed", actual)
    return observations


transfer_observations = run_transfer(transfer_weighted_fuse, TRANSFER_CASES)
TRANSFER_PASSED = all(r["passed"] for r in transfer_observations)
print("TRANSFER_STATUS", "PASS" if TRANSFER_PASSED else "NEEDS_WORK")

Search for a weight, and do not fool yourself

When your fusion passes, the cell below fuses BM25 with all-minilm for several weights on BM25 and evaluates each. Look for a weight that does at least as well as all-minilm alone on both kinds of question.

Then answer honestly: you chose the weight by looking at these twelve questions. How would you check that it still helps on questions you have not seen?

python

if TRANSFER_PASSED and connected:
    for weight in (1.0, 0.5, 0.25, 0.1):
        for kind in ("direct", "paraphrase"):
            chosen = [(q, answers) for q, answers, k in QUESTIONS if k == kind]
            fused = [
                transfer_weighted_fuse([bm25_ranking(q), minilm_ranking(q)], [weight, 1.0], 12)
                for q, _ in chosen
            ]
            scores = evaluate(fused, [answers for _, answers in chosen], 2)
            recall, rank = scores["recall_at_k"], scores["reciprocal_rank"]
            print(f"BM25 weight {weight:<4} {kind:10} recall@2 {recall:.3f}  reciprocal {rank:.3f}")
else:
    print("Complete evaluate and the fusion first.")

Save your evidence and explain the result

Fill in the prediction notes and your explanation before saving. Include:

the exact observed value, and the input that caused it;
your code's invocation point;
one failed hypothesis;
the strongest claim the evidence still cannot support.

Twelve questions are few. A difference of one question moves a recall by 0.167. Chapter 15 shows how to put an interval around a rate measured on so few cases.

python

explanation_notes = {
    "causal_trace": "Explain the input, learner invocation and observed result.",
    "failed_hypothesis": "Describe a prediction the evidence changed.",
    "remaining_limit": "Name the guarantee not established by this experiment.",
}
course_submission = {
    "unit": "ch06-b",
    "planned_minutes": 90,
    "starting_evidence": HANDOFF_ORIGIN,
    "prediction": prediction_notes,
    "explanation": explanation_notes,
    "core_report": exercise_report,
    "measurement": connected,
    "transfer": transfer_observations,
    "explanation_review": "HUMAN_REVIEW_REQUIRED",
}
submission_path = COURSE_WORK / "ch06-b-submission-v1.json"
submission_path.write_text(
    json.dumps(course_submission, indent=2, sort_keys=True), encoding="utf-8"
)
print("Saved evidence:", submission_path)
print(
    "COURSE_REPORT="
    + json.dumps(
        {
            "unit": "ch06-b",
            "transfer_passed": TRANSFER_PASSED,
            "starting_evidence": course_submission["starting_evidence"],
            "edition": "student",
        },
        sort_keys=True,
    )
)

Keep building with Prof Rod

Found this material through a colleague, classroom or shared download? Get the complete book at profrod.ai/book and join the Prof Rod learner community. Bring one result, one question or one failure you learned from. Share this resource with another learner and keep its source links with it so they can find the full course and future updates.

Embeddings and Vector Search: Unit B Exercises

Run the self-contained setup

Commit to a prediction before the examples

Two metrics, by hand

What each retriever does with one paraphrase

Choose an explicit starting point

2. Construct evaluate

3. Measure three retrievers on Lucy's questions

4. Save the measurement

Changed-constraint construction: weighted fusion

Search for a weight, and do not fool yourself

Save your evidence and explain the result

Keep building with Prof Rod

2. Construct `evaluate`