The 2x2 for Agent Design: Workflows, Agents, and Per-Layer Evals
Most agent designs answer two questions by accident: who waits for the result, and who decides the next step, the model or your code. This course makes you answer them on purpose, with a 2x2 and the cost of each quadrant: reliability as p^n, latency across sequential calls, input tokens that grow with the square of a loop's length. The running example: evaluate every new checkpoint overnight and explain any regression by morning, split into four layers, each with its own evaluation (Wilson intervals, paired comparisons, a grader checked against human labels). It closes with a seven-step answer to a system-design interview's 'design an agent that...'.
MethodThis course teaches a discipline. The examples use today's tools; the method is meant to outlast them.
Rod Rivera
Professor
Desk
Every lesson is free. Lessons marked open need no account; the rest open once you register.
What You'll Learn
- Separate who waits for the result from who decides the next step, and place any layer of a design on the 2x2 that crossing them produces
- Derive what each quadrant costs, with the arithmetic shown, in reliability (p^n and recovery), latency (sequential calls) and tokens (n·P0 + Δ·n(n−1)/2)
- Say when a workflow beats an autonomous agent, and read an open agent loop and a routing workflow the way a design reviewer would
- Decompose a one-sentence design question into layers, derive a budget for each, and type the handoffs between them with contract tests
- Walk a design through the lethal trifecta, and harden a model-planned layer with a sandbox, a derived budget, a tool allowlist and observability
- Give every layer its own eval loop, with Wilson intervals, paired McNemar comparisons, graders measured by Cohen's kappa, and offline, online and regression loops
- Answer "design an agent that..." in seven steps, with the numbers and the eval plan a reviewer or interviewer will ask for
Prerequisites
- Comfort calling a model API with tools; this course does not re-teach tool-calling mechanics from scratch
- Basic probability (independent events, a binomial proportion); every formula used is derived or linked to its derivation in the book
- Enough Python to read and run short standard-library listings; they run on Python 3.12 in Colab
- Helpful, not required: the book's chapters on the agent loop (/book/ch03-agent-loop) and evaluation (/book/ch16-evaluation)
Syllabus
Read the rest of The 2x2 for Agent Design: Workflows, Agents, and Per-Layer Evals
The first lessons need no account. Sign in free to read the rest and keep your place on any device.
