The Question Nobody Asks Before Building

Before you start

Prerequisite: none. This is the course's opening lesson. After this lesson, you can: state the running design question of this course, split it into the layers it contains, and explain why swapping in a stronger model does not repair a design whose layers sit in the wrong quadrant.

A design question, stated plainly

Design reviews and system-design interviews for applied-AI work tend to open the same way: a one-sentence problem, and a blank whiteboard. This course runs on one such problem from start to finish.

Design an agent system that evaluates every new model checkpoint overnight, and explains any regression by the time the team arrives in the morning.

The reason to practice on this one is that it is ordinary lab work. Checkpoints arrive from a training run. Each needs a fixed set of evaluations, a results table with honest error bars, and, when a number drops, someone to find out why. None of it is exotic. All of it has to be right, because a team will decide what to train next from that table.

The tempting first answer is a single agent: give a model tools to launch evals, read results and search logs, and let it work through the night. Hold that answer for a moment. It makes two decisions by accident.

The two questions a single agent answers by accident

Who is waiting for this to finish?

Is a person at a prompt, reading each turn? A researcher who opens a report at 9 a.m.? Or nobody at all, only a later job that reads a table?

Who decides what happens next: the model, or your code?

Does the model choose its next tool call, turn by turn? Or does your code hold the path, and call the model only for one bounded step, such as grading a free-form answer against a rubric?

A single overnight agent answers "nobody is waiting" and "the model decides" for every part of the system at once. Some parts of the checkpoint system fit those answers. Others do not, and they fail in predictable ways when forced into them.

These two questions are independent. A system can have a person waiting whether or not the model holds the route, and the model can hold the route whether or not anyone is waiting. That independence is what turns two questions into a 2x2 grid, which lesson 03 builds.

Two independent questions, not one maturity ladder. Who is waiting?a person now vs. nobody until later; Two independent axes,not a good-to-bad scale; Who decides the next step?the model vs. your code; The 2x2 that lesson 03builds from these answers — Two independent questions, not one maturity ladder

The four layers inside one sentence

Read the design question again and list the distinct pieces, where a piece is anything that needs its own answer to both questions.

Layer	What it does	Who waits
A. Eval request desk	A researcher types "run the reasoning suite on checkpoint 4120 against 4080, 500 samples"; the system confirms the exact run it will launch	The researcher, for a few seconds
B. Nightly scoring pipeline	Runs the fixed suites on each new checkpoint, grades answers, writes the results table	Nobody until morning
C. Regression investigator	When a comparison flags a drop, reads failing transcripts and past notes, drafts an explanation	Nobody until a researcher reads the draft
D. You, building it	You in a coding agent, editing the harness, graders and prompts	You, right now

Four rows, and already three different answers to "who waits." The "who decides" column differs too. Layer A has a small, known set of valid requests, so code can hold the route. Layer B runs the same steps every night. Layer C cannot be scripted in advance, because each regression sends the investigation somewhere new. Layer D is open-ended by nature.

Here is what goes wrong with the single-agent answer. Put Layer A inside the overnight agent's loop, and a researcher who asks for one eval run now waits while a model plans its way through several tool calls to do something a validated command could do in one. Worse, the model may launch a slightly different run from the one requested: another sample size, the wrong baseline. That is not a style problem. It is a wrong row in a results table that someone will cite.

Why a stronger model is the wrong first fix

When a design like this disappoints, the usual proposal is a stronger model. Sometimes that helps a single step. It does not change the structure. A stronger model still makes the researcher wait through a loop that should have been one call. It still leaves the launch of a GPU job to a sampled decision when the valid options were known in advance. And the arithmetic of a multi-step loop still compounds: lesson 03 shows that a loop of 12 model decisions, each right 97% of the time, finishes all 12 correctly about 69% of the time, while a single validated step at the same 97% stays at 97%.

The fix is to ask the two questions per layer before choosing any tool. That is the move this course trains, and it is the move a reviewer is listening for.

Quick check — A checkpoint-eval agent sometimes launches runs with the wrong baseline when a researcher asks for one. The proposed fix is a stronger model. What does this lesson suggest is missing from that diagnosis?

Quick check — You are asked to design an agent that answers questions from researchers about the results of the nightly evals, and pages the on-call researcher if a headline metric drops. How many distinct layers should you look for before choosing an architecture?

Where this course goes next

The next lesson names the parts every agent shares, and prices them: how the context window fills, and why the tokens of a loop grow with the square of its length. Lesson 03 crosses the two axes into the 2x2 and derives what each quadrant costs. From there, every lesson returns to the checkpoint system, and every layer gets an evaluation of its own.

Continue to Lesson 02

The anatomy every agent shares, and the token arithmetic that follows from it.