Why Agent Systems Fail, and How a Lab Would Find Out
An agent that passes its demo can still fail most real tasks, usually for arithmetic reasons: reliability compounds as p to the n, pass@k and pass^k answer different questions, and a one-run benchmark gap can sit inside its own noise. You read and write postmortems, sort failures by harness layer, and build the control four agent shapes share: an evidence gate, deterministic code that refuses a claim without its evidence. The capstone is an interview work sample: a manifest, a verifier, a human gate, one defended failure mode and an evaluation with error bars. The companion lab in the public profrodai-resources repo runs offline on Python's standard library.
MethodThis course teaches a discipline. The examples use today's tools; the method is meant to outlast them.
Rod Rivera
Professor
Desk
Every lesson is free. Lessons marked open need no account; the rest open once you register.
What You'll Learn
- Derive why per-step reliability compounds (p to the n) and what a retry with a verifier changes
- Tell pass@k from pass^k, and explain why a measured pass^k sits above the independence prediction
- Put a binomial error bar on an agent success rate and judge whether a benchmark gap is signal
- Read a public postmortem for symptom, impact, root cause, the eval that missed it, and the fix
- Write a failure analysis in the shape an interview loop or a take-home work sample can probe
- Sort agent failures by harness layer (tools, state, retries, context, evaluation, oversight)
- Build an evidence gate, a deterministic verifier that refuses a claim without its source
- Check a model grader's calibration bucket by bucket, with an interval on every bucket
- Explain why a policy-enforced conversational agent keeps decisions out of the language model
Prerequisites
- Comfort reading Python and YAML
- Basic probability (independent events, a binomial proportion); the course derives the rest
- No prior agent-building experience required; the book's agent loop chapter is the companion
Syllabus
Read the rest of Why Agent Systems Fail, and How a Lab Would Find Out
The first lessons need no account. Sign in free to read the rest and keep your place on any device.
