Measure Stale Agent Memory as Recall@k, Then Retire It by Source Revision

The note that keeps winning

A research agent answers questions from a memory built from weeks of experiment logs. For a month, the logs said the math eval runs 4-shot, and the agent stored that observation many times: in run summaries, in debugging notes, in a comparison table. Then the eval harness changed to 0-shot in revision v18, and one new note records it.

Someone asks, "How many shots does the math eval use?" The store embeds the question and returns the nearest notes. Most of them say 4-shot, because most of the notes about the math eval were written before the change. Every one is a strong semantic match. None is current.

The retrieval did its job. It found notes about the math eval's shot count. It was never asked whether they are still true.

Measure it before you fix it

This is a retrieval-quality problem, so measure it with retrieval metrics. Chapter 5 derives recall@k and reciprocal rank; two follow directly for staleness:

• stale@1: the share of queries whose top-ranked note is superseded. It is the rate at which a top-1 answer would be wrong.
• recall@k of the current note: the share of queries whose current note appears in the top k. For an agent this is the one that matters: a current note outside the top k never reaches the model.

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & relevant) / len(relevant)


def stale_at_1(ranked, superseded):
    return ranked[0] in superseded


# One query, "how many shots does the math eval use?", ranked by some embedding model.
ranked = ["obs-03", "obs-11", "obs-07", "harness-v18", "obs-01"]  # obs-* say 4-shot (superseded)
current = {"harness-v18"}  # says 0-shot, from the harness's current revision
superseded = {"obs-01", "obs-03", "obs-07", "obs-11"}

print("recall@3 unfiltered", recall_at_k(ranked, current, 3), "stale@1", stale_at_1(ranked, superseded))
filtered = [d for d in ranked if d not in superseded]
print("recall@3 filtered  ", recall_at_k(filtered, current, 3), "stale@1", stale_at_1(filtered, superseded))

# If m stale copies and the one fresh note score about alike, the fresh note's rank is
# uniform over 1..m+1, so P(fresh in top k) = k / (m + 1).
for m in (1, 4, 9):
    print(f"m={m}: P(fresh in top 3) = {min(1, 3 / (m + 1)):.2f}")

recall@3 unfiltered 0.0 stale@1 True
recall@3 filtered   1.0 stale@1 False
m=1: P(fresh in top 3) = 1.00
m=4: P(fresh in top 3) = 0.60
m=9: P(fresh in top 3) = 0.30

I ran it with Python 3.12 on September 30, 2026. The ranking is a hand-written example, not a model's output. The last three lines are arithmetic, and they are the useful part. Suppose the old fact was stored m times and the new one once, and all m + 1 notes are about equally similar to the question. Then the current note's rank is equally likely to be any of 1 through m + 1, and the chance it lands in the top k is k / (m + 1). With nine old observations and k = 3, that is 3 / 10 = 0.30. Repetition, which feels like evidence, crowds the current fact out.

The assumption of equal similarity is a simplification. The new note may be phrased closer to the question, or further from it. That is exactly what a measurement with a real embedding model settles.

The protocol, with real embeddings

This article reports no measured numbers. Here is the experiment, sized so it runs in a Colab notebook on a small open embedding model; Chapter 6 measures all-minilm and shows how to store each vector with the model that made it.

What you expect, stated as a hypothesis to test rather than a result: unfiltered stale@1 rises with m; filtered stale@1 is zero by construction; filtered recall@3 is limited only by how well the current note matches the question, which is Chapter 6's paraphrase problem.

Retire by revision, keep the history

The fix is not "delete old memories more often." Deleting loses the record of what the team believed and when, and it can throw away something still true. Give every note enough structure that a program, not a similarity score, decides whether it may answer a present-tense question.

When eval.math.shots moves from v17 to v18, the v17 notes become superseded, and a new note is written at v18 with status: current. Current-answer retrieval filters on status and cross-checks the revision against the source's live revision. The old notes stay, so "what did we assume in August" still has an answer. Chapter 6's vector index does the same in its "keep only the current revision" section: search reads only current rows, and the superseded row stays on record.

View full-size diagram

Scope the invalidation to what changed

Wiping everything the agent knows on every source change is the opposite mistake. If the harness revision bumps because the math eval's shot count changed, the code eval's timeout did not move, and re-verifying it wastes work and can leave a window with no answer at all.

Scoped invalidation needs a per-fact revision or a trustworthy change list. If v18 is only a global revision of the harness config, a mismatch says something changed, not what. In that case revalidate every fact that depends on that source, or get the change list first. Do not claim fact-level precision from a global version number.

View full-size diagram

Someone has to own the trigger. Leaving it to "the agent will notice from context" is how stale notes survive. A small ownership record per fact domain is enough: fact_id, authoritative_source, updater, invalidation_trigger (for example, on every harness config commit) and escalation_when_missing. Giving every memory an owner and an expiry rule works through ownership and conflict escalation in detail.

Check the decision

Why this matters in a lab

Retrieval evaluation is core applied-AI work, and memory staleness is a retrieval failure that ordinary benchmarks do not measure, because their corpora do not change under them. An agent that works in a research repository sees its sources change constantly: configs, harnesses, datasets, baselines. Being able to say "our memory's stale@1 is this, with this interval, and the revision filter takes it to zero at this recall cost" is the kind of claim a team can act on.

In an interview, the derivation above is a good test of whether you think in distributions: why does repetition hurt, what is the probability the current note makes the top k, and what assumption makes that formula exact? The Tools, Memory, and Multi-Agent Systems course covers the surrounding memory architectures.

What would prove this wrong

The revision filter fails if the source has no reliable revision and you fall back on recorded_at: a timestamp says when a copy was written, not whether anything changed upstream afterward. Use a content hash, a re-fetch, or a documented freshness interval instead, and say which.

Two writers can race to publish different current values; the store needs a uniqueness rule on the current pointer and a conditional update, which an in-memory example does not test. And the recall argument fails if, on your model and your notes, the current note outranks the old ones regardless of m. Run the protocol and find out.