Run It Logged Out Before You Polish the Plots

A notebook can produce clean figures for you and fail on its first cell for anyone else. The difference is usually state you carry and a new reader does not: an API key exported in your shell, a model cache on disk, a variable defined in a cell you later deleted.

Test the reader's starting state before you spend time on the plots. The Claude Code best-practices guide uses this exact case as its example of a well-scoped test request: "covering the edge case where the user is logged out."

Give Claude Code two concrete journeys to run against your eval script or notebook:

1. Logged out. Run with the credential removed, for example env -u OPENAI_API_KEY uv run python run_eval.py, where run_eval.py is your own entry point limited to a few items. Expected: an early, readable error that names the missing key. Not a stack trace from deep inside the client, and not a silent fallback to cached results from last week.
2. Clean kernel. In Colab or Jupyter, restart the runtime and run every cell in order. Expected: the numbers you reported, top to bottom, with no manual steps in between.

Then repeat with the key set, so you know both paths work.

The silent fallback is the one to hunt for. A script that reads cached outputs when the API call fails produces a results table that looks current and is not. Have the script record where each result came from, live call or cache, and print both counts at the end. If the cache count is not zero on a run you believe was live, stop there.

Use a key scoped to a test project and a small item limit so the exercise costs little. What a model call is sets up the client and the key handling this check assumes.

Claude Code documentation.