The passage the answer never mentioned
A lab runs an internal research assistant over its document store: the cluster wiki, experiment notes and eval reports. Most of it is open to everyone. Some is not. An eval report on an unreleased model is restricted to the eval team. Ana is on that team. Bo is not.
Bo asks, "What did the latest eval show?" The pipeline embeds the query, pulls the top candidates by vector similarity, reranks them, and hands the best few to the model. The model writes a careful answer about Bo's own notes and never mentions the restricted report. Did the system behave?
You cannot tell from the answer. The question is what entered the context window. If the restricted report was among the reranked passages, the boundary was crossed the moment those bytes went into the prompt. A model choosing not to repeat something is not a permission check, and nothing guarantees it will choose the same way on the next call.
Relevance and authorization are different filters
A retrieval pipeline has three relevance stages: embed the query, search an index for near neighbors, and rerank the candidates with a model that scores each query-passage pair. Chapter 6 builds the first two from scratch and measures them. NVIDIA's NeMo Retriever reranking documentation describes the third as a relevance-scoring stage. None of the three asks whether this caller may see this passage.
So authorization is its own filter, driven by its own data: who is asking, established by the authentication layer, and which documents that caller may read right now, from an authoritative permission source. A valid access token does not answer the second question. Token validation says the request may reach the service; it does not map a caller to documents. Checking the caller's authority, not the service account's covers the identity half.
Where the filter sits
Constrain the search itself to documents the caller may read, then recheck the IDs before anything is sent to a reranker or a model. If your vector store supports an enforced filter on the query, use it. A hosted reranker that receives a passage has already received it; filtering after that is too late for the reranker boundary.
Here is the rule as a fixture. The documents carry obvious markers so the test can inspect the prompt bytes, not only the IDs:
from dataclasses import dataclass
@dataclass(frozen=True)
class Doc:
doc_id: str
text: str
# Synthetic documents with obvious markers, never real restricted material.
DOCS = [
Doc("wiki-cluster", "How to request GPUs on the shared cluster."),
Doc("eval-restricted", "MARKER-EVAL-TEAM-ONLY unreleased benchmark scores."),
Doc("bo-notes", "MARKER-BO-ONLY draft notes on the tokenizer ablation."),
]
PERMISSIONS = { # the authoritative source, read at request time
"ana": {"wiki-cluster", "eval-restricted"},
"bo": {"wiki-cluster", "bo-notes"},
}
def allowed_ids(user):
if user not in PERMISSIONS:
raise PermissionError("authenticated caller required")
return PERMISSIONS[user]
def search(query, docs, k):
return docs[:k] # stand-in ranking: every doc is "relevant" in this fixture
def build_context(query, user, k=3):
allowed = allowed_ids(user)
candidates = search(query, [d for d in DOCS if d.doc_id in allowed], k) # filter before ranking
context = [d for d in candidates if d.doc_id in allowed_ids(user)] # recheck before the prompt
return context
bo = build_context("what did the latest eval show?", "bo")
print([d.doc_id for d in bo])
prompt = "\n".join(d.text for d in bo)
assert "MARKER-EVAL-TEAM-ONLY" not in prompt
assert {d.doc_id for d in bo} <= allowed_ids("bo")
print("no restricted marker in Bo's prompt bytes")['wiki-cluster', 'bo-notes']
no restricted marker in Bo's prompt bytesI ran it with Python 3.12 on September 30, 2026. It has no embedding model and no vector store; the ranking is a stand-in. What it shows is where the checks go and what the assertions look at: a subset check on IDs against current permissions, and a search for the forbidden marker in the actual prompt text. An ID check alone cannot catch text attached to the wrong ID.
The fixture shows the filtering logic. It does not show that your vector store applies its filter correctly, that your reranker adapter forwards only what it is given, or that a shared cache respects the boundary under load. Run the same marker test through the deployed path.
Two metrics, and they are not symmetric
Permission-aware retrieval has a safety metric and a quality metric, and they behave differently.
Leakage is the number of (query, caller) pairs in which any unauthorized passage reached the context. It is not estimated with an interval, because the target is zero and a single case is a failure. Test it exhaustively over your fixture: every test caller, every query, with a marker in every restricted document, and assert on the prompt bytes.
Recall@k is the share of relevant, authorized passages that appear in the top k. Chapter 5 derives it with precision@k and reciprocal rank, and names why it matters most for agents: a relevant record outside the top k never reaches the model. The filter can cost recall, and how much depends on where it runs.
What post-filtering costs
The tempting design is to search the whole index for the top K, then drop what the caller may not see. Suppose the caller can read a fraction f of the index, and suppose, as a first approximation, that permission is independent of relevance. Then the number of authorized passages among the top K is binomial with mean f·K.
Take f = 0.05 and a context that needs k = 5 passages. The mean says K = 5 / 0.05 = 100. But the mean is not enough: with K = 100 the chance of getting at least five authorized passages is only 0.564. The smallest K that gets five with 95% probability is 181.
from math import comb
def p_at_least(k, K, f):
return 1 - sum(comb(K, i) * f**i * (1 - f) ** (K - i) for i in range(k))
for K in (100, 150, 200):
print(K, round(p_at_least(5, K, 0.05), 3))
K = 5
while p_at_least(5, K, 0.05) < 0.95:
K += 1
print("smallest K with 95% chance of 5 authorized:", K)100 0.564
150 0.874
200 0.974
smallest K with 95% chance of 5 authorized: 181That is arithmetic under an assumption, not a measurement of any system. In real stores permission and relevance are correlated: people mostly search their own team's documents, which helps; a restricted report can also be the nearest neighbor for exactly the query that matters, which hurts. The number tells you the direction: post-filtering makes recall depend on how much of the index each user can see, and the users who see least lose the most. Pre-filtering inside the search keeps k authorized results by construction, at whatever cost your index charges for filtered search.
The measurement protocol
This article reports no measured recall. Here is how to get yours.
A few hundred documents, each with an access list, a unique marker string in each restricted one, and 30 or more queries per test caller with their relevant authorized documents labeled in advance.
For every caller and query, capture the reranker input and the final prompt, and assert that no marker from a document outside that caller's current permissions appears. The pass condition is zero, with no interval.
Pre-filter inside the search, and post-filter a top-K list for two or three values of K. Report recall@5 per caller group, with a Wilson interval on each proportion, since each query's hit or miss is a trial.
Revoke one document from one caller, rerun without rebuilding the index, then serve a cached candidate list to a different caller. Both must still show zero leakage.
The two usual bypasses
Caches. An answer or candidate list cached for Ana under a query-only key must not be served to Bo for the same query. Include the caller and an authorization revision in the key, and still recheck current permissions before the passages leave the retrieval service. Invalidating cached answers when authority changes works through the revision design and the race with an in-flight request.
Stale index metadata. If permissions are copied into the index at ingestion time, a revocation after indexing leaves the old grant in place until the next rebuild. Read permissions from the authoritative source at query time, or version them and refuse results whose version is stale. If the permission source is unavailable, deny; do not treat a failed lookup as unrestricted access.
Check the decision
Why this matters in a lab
Retrieval over internal documents is one of the first agent systems many applied-AI teams build, for their own staff and for customers. An interviewer who asks you to design one is likely to probe two things: whether retrieval quality is measured (recall@k on a labeled set, not a demo) and whether access control is enforced before the prompt (not by instruction). The strong answer names both metrics, says why leakage is tested exhaustively and recall statistically, and explains what post-filtering costs with the arithmetic above.
What would prove this wrong
The filter has failed if any of these happen in your test: an ID outside current permissions appears in the search output, the reranker input or the final context; a restricted marker appears in the prompt bytes; a cached list for one caller is served to another without the recheck; or a revoked document still surfaces because the filter trusted index metadata. The recall argument fails if permission and relevance are so correlated in your data that post-filtering loses nothing. Measure it before you assume either way.
Build word vectors from scratch, then measure a real embedding model against BM25 on paraphrased questions with recall@3.
