AI Cache Authorization: When Answers Outlive Access

Suppose your retrieval system answers a question correctly, caches that answer, and then someone revokes the asker's access to the document the answer came from. The question text hasn't changed. The document hasn't changed. Only one thing changed: whether this particular person is still allowed to know what's in it. If your cache doesn't know that fact changed, it will cheerfully hand back the old answer to the same person who asked the same question, and it will be wrong in a way that no amount of re-embedding or re-ranking would ever catch, because the retrieval was correct. The authorization was not.

This is the puzzle worth sitting with before writing any caching code: caching is usually justified by "the input didn't change, so the output shouldn't need recomputing." That logic holds for prompt caching, where a provider reuses the computation behind a repeated block of input tokens. Anthropic's prompt caching documentation describes exactly that scope: caching concerns reuse of input computation, and current pricing, eligibility and latency behavior are documentation details you check at the time you build, not facts to assume forward (Anthropic, prompt caching documentation). The provider may isolate caches by account or other documented boundaries. That does not enforce your application's per-user document policy. The moment you cache a finished answer keyed only on the question, you've quietly promoted a computation-reuse trick into an authorization decision, and those are different problems with different failure modes.

Two caches that look alike and aren't

Prompt caching and answer caching sit at different layers, and confusing them is the misconception this article exists to correct. Prompt caching, as Anthropic documents it, saves you from repaying the cost of re-processing a long, unchanged prefix of input tokens across calls. It's a provider-side optimization on computation. Answer caching is something you build: you store the finished response to a query so that a repeated query doesn't have to re-run retrieval, re-rank passages, or re-invoke a model at all. The provider's cache has no opinion about who's allowed to see the finished text. Your application must enforce that policy on the response path, including cache hits. It can do so through current access checks, revision-aware cache lookup, or both.

That distinction matters because the natural cache key for an answer cache is "the query, maybe normalized." It's the same shape a search engine uses. For a public search engine, that's fine, because everyone gets the same answer. For a permission-sensitive system, that key is a bug waiting for someone to revoke access. If your key doesn't include who's asking and what they're currently allowed to see, the cache has no way to know that the ground under the answer shifted.

A worked case: three documents and one revoked reader

Take a small retrieval fixture, the kind used earlier in this course's own retrieval-pipeline teaching material (see the two-stage embed-then-rerank lesson at /courses/nvidia-nim-in-production). Three documents exist. One of them, call it d3, is private and visible only to a reader named Ana in this scenario. Ana asks a question whose best answer draws on d3. The system embeds her query, retrieves and reranks candidates, generates an answer, and caches it.

Now imagine Ana's access to d3 is revoked, and she asks the identical question again, word for word. If the cache key was hash(normalized_query), the system finds a hit and returns the cached answer immediately, no retrieval, no authorization check, nothing. The text of the question and the text of the document are both unchanged. Only Ana's standing to receive that answer changed, and a key that never encoded her standing has no way to notice.

The fix is to widen the key so that authorization state is part of what identifies a cache entry, while retaining a current authorization check before release. A workable key, proposed here as a design to build and test rather than a measured result, looks like a five-part tuple:

With authorization_revision in the key, revoking Ana's access to d3 means incrementing her revision counter. The old cache entry, keyed to revision 7, simply stops matching. An entry stored only at revision 7 cannot satisfy the revision-8 key. Retrieval must then enforce current permissions; a cache miss alone does not remove d3 from a permissive retriever. No code has to remember to hunt down and delete the old entry. It becomes unreachable through this lookup if every path uses the current authoritative revision. Retention or deletion rules may still require purging old bytes. Revision reads from stale replicas can break the assumption.

Before serving a stored answer, compare its dependency versions with the authoritative document versions. The document set may be known only after retrieval, so one implementation looks up a candidate entry by principal, authorization revision and query, then validates its stored dependency list. Include tenant identity and other policy inputs where they affect access.

The part a static test misses: revocation mid-flight

A before-and-after check, ask, revoke, ask again, is necessary but not sufficient, because it only tests revocation that completes before the next request starts. Real systems have requests in flight when access changes. Picture Ana's second request already past retrieval, holding a generated answer, and about to be written to the cache and returned, at the exact moment an administrator revokes her access and bumps her authorization revision from 7 to 8.

A response object mid-flight, entirely hypothetical for this teaching case, might look like this:

pending_response = {
    "operation_id": "q17",
    "principal_id": "Ana",
    "auth_revision": 7,
    "document_revisions": ["d3:v2"],
    "answer": "fixture answer",
    "state": "pending",
}

# Authoritative store, checked immediately before release:
current_auth_revision = 8  # bumped by the revocation

if pending_response["auth_revision"] != current_auth_revision:
    outcome = "authorization_changed"
    # Do not return this answer. Do not write it to the cache
    # under the new revision. Discard or hold for audit.
else:
    outcome = "release"

Running this comparison in Python is a two-line if, but the discipline it represents is the whole point: the check has to happen immediately before the answer is released or stored, not at the moment the request started. If the system checked authorization only once, at the top of the request, this exact scenario slips through, because at that moment Ana still had access. The revocation happened during the work, and the recheck catches this ordering. It still leaves a race if revocation can commit between the check and release.

Define the ordering guarantee explicitly. The authorization store and response-admission step need a shared transaction, lock or equivalent coordination point. A response admitted before revocation may already have bytes in transit; no later permission change can retract bytes the caller has received. Streaming responses need their own policy. The contract here concerns admission after a committed revocation.

A companion positive control matters just as much as the failure case: if auth_revision had stayed at 7 throughout, the response should complete normally. A test suite that only ever exercises the revoked path can create a system that's paranoid in the wrong direction, refusing to serve anyone anything. You want both outcomes proven: revoked requests discarded, unrevoked requests served.

There's a second failure mode worth naming here, distinct from the first: does discarding Ana's stale response accidentally touch anyone else's cache entries? If invalidation logic walks the cache looking for "anything related to d3" and deletes broadly, a completely unrelated caller with valid access to d3 loses their good cached answer for no reason. The fix is the same key discipline: invalidation should operate on the specific (principal_id, authorization_revision) pair going stale, not on the document identifier alone, precisely when the change affects only Ana. A document deletion or document-wide policy change may legitimately invalidate every dependent answer; do not scope that broader change to one principal.

Who owns which piece of this

This is a design decision with an operating owner, not just a coding pattern. Someone has to be the authority that increments authorization_revision, and that has to be the same system of record that actually grants and revokes access, not the caching layer inferring it secondhand. The Model Context Protocol's authorization specification is useful here for a distinction worth holding onto: it treats authorization as resource-bound and separate from the transport credential carrying a request, which means validating a token for a resource does not establish the caller's current access to every internal document (Model Context Protocol, authorization specification). Your cache key needs the authorization fact, not just the credential that got the request through the door.

Practically, that means a named owner for the revocation event itself: when access changes, something must reliably bump the revision counter, and something must define what happens to a request that's already past that checkpoint when the change lands. Left undefined, that's exactly the mid-flight gap the worked example above walks through. Defined explicitly, it's a short contract: "revocation increments the principal's authorization_revision synchronously with the access change, and any response not yet released is rechecked against the current revision before release or storage."

Check the decision

What would prove this design wrong

State the failure condition plainly so you know what to test for: if a fresh request, issued after revocation with no in-flight timing involved at all, still returns the old answer, the key design has failed, because a clean miss should be structurally guaranteed once the revision changed. That's the first thing to check, and it's the cheap, deterministic case. The harder, more valuable test is the race: pause a request after it has retrieved data but before it returns, revoke access, resume the request, and confirm the recheck catches it. If that recheck only happens at the very start of request handling, the race will slip through even though the simple before-and-after test passes. Both tests belong in the suite, because they catch different bugs, and a system that passes only the easy one hasn't earned confidence yet.

None of this has been run against a real shared cache store as evidence for this article; it's a design worked through in full so it can be built and checked, not a claim of a measured result. The existing retrieval-pipeline lesson referenced above shows a related but separate lesson, that hardcoded model identifiers in a retrieval pipeline eventually break when a vendor retires them. That's a different kind of staleness, caught by a version check against a live catalog. This article's staleness is about who's allowed to see an answer, caught by an authorization revision in the cache key. Keep the two straight: one is about whether the pipeline still works, the other is about whether it's still allowed to answer.