Evaluation Engineering Across Both Quadrants

Sarah's Layer C drafted a reply last week that was factually correct, well-written, and recommended a competitor's product by name. Nothing crashed. No error fired. The system did exactly what it was asked to do: draft a helpful reply. It just wasn't the reply anyone wanted, and nothing in Sarah's test suite caught it, because Layer A style tests ("did the right slot get filled") don't apply to freeform text, and Layer C had no test at all for "does this draft mention a competitor."

That gap is what this lesson is about. Lesson 8 hardened Sarah's system against being attacked and against quietly burning through budget. Neither discipline answers a more basic question: is the system doing its job? For a deterministic program, that's yes-or-no: the test passes or it doesn't. For a probabilistic system, "did it work?" stops being a yes/no question and becomes a distribution. Most of it will work most of the time. The engineering problem is knowing which part, how often, and finding out before a customer does.

That's evaluation engineering: the one craft that spans every quadrant in this course, not just the stochastic one. This course's own view is that the most distinctive skill an agent engineer has isn't prompt-writing. It's building the measurement systems that tell you whether the non-deterministic core of a system is behaving, before the bill, an incident, or the customer complaint tells you for you.

What "eval" means, layer by layer

Each of Sarah's four layers needs a different eval method, because each one fails in a different shape:

• Layer A (CALM chat): conversation-level replay tests, slot assertions, flow completion rate per intent.
• Layer B (extraction pipeline): schema validation rate, field-level precision and recall against a labeled set.
• Layer C (research agent): LLM-as-judge on a held-out set, with regression suites of seed prompts and known-good outputs.
• Layer D (Sarah in Cursor): Sarah's own judgment, informed by what the eval results from A, B, and C are telling her.

Notice the shape of that list: it gets less structured moving from A to D, the same order the trifecta risk got worse in lesson 8. That's not a coincidence. The less deterministic a layer is, the more its evaluation has to lean on judgment rather than exact-match assertions, and the more disciplined that judgment needs to be to stay honest.

A concrete starting point

"Build eval suites" is useless advice without a template, so here's the actual starting point: write ten representative inputs, write the correct output for each, run them on every deploy, and fail the deploy if any of them regress. That's it. Everything past that is refinement.

tests:
  - name: "enterprise_blocking_issue_pages_oncall"
    user_messages:
      - "my production is down and I'm losing money"
    slot_assertions:
      issue_category: "technical"
      urgency: "blocking"
      account_tier: "enterprise"
    flow_assertions:
      - action_page_oncall was called
      - utter_confirm_routing contained "on-call engineer"

  - name: "billing_question_non_blocking_routes_to_queue"
    user_messages:
      - "I think my invoice is wrong, no rush"
    slot_assertions:
      issue_category: "billing"
      urgency: "non_blocking"
    flow_assertions:
      - action_route_to_queue was called
  # ... 8 more

Run these on every deploy. Not sometimes: every deploy. The eval suite is the only thing standing between a routine model update and a silent regression nobody notices until a customer does.

LLM-as-judge: the trap and the escape

Layer A's assertions check exact values: a slot, a called action. Layer C's output is free-form text: a draft reply, a summary, an explanation. You can't check that for equality against a known-good string, so the standard move is LLM-as-judge: ask a second model whether the output meets a set of criteria, and use its verdict as the test result.

The trap is that judges carry their own biases. They favor longer outputs over correct ones (verbosity bias). They favor outputs from their own model family (a GPT judge tends to prefer GPT-written text). They favor confident-sounding text over accurate text. And judge verdicts are themselves non-deterministic. The same output, scored against the same criteria, can get a different verdict on a different run.

The most valuable test in any suite is still the one where a human labeled the ground truth. Judge-based tests are a scaling mechanism for extending that human judgment across volume, not a substitute for having made it in the first place.

Three eval loops, not one

Offline, online, and regression eval answer different questions, and conflating them is how teams end up with a false sense of coverage.

Offline eval runs a fixed test set on every deploy: deterministic inputs, known-good outputs, a pass/fail verdict. It's the gatekeeper between a dev branch and main. For Sarah, that's the ten representative tickets per layer, run in CI.

Online eval scores a sample of real production traffic in the background, with no customer impact, to catch drift: are outputs getting worse on tickets that look like X. For Sarah, every hundredth ticket gets its Layer C draft scored by an LLM judge against the rubric, with an alert if the average score drops more than 10%.

Regression eval turns every production incident into a permanent test case. When something goes wrong, the failing input joins a regression set that runs on every deploy forever. This is how a suite earns its keep over time rather than staying frozen at day-one coverage. For Sarah, the first time Layer C drafts a reply recommending a competitor's product, that exact ticket goes into the regression set with the assertion that output must never mention a competitor again. Every deploy checks it, permanently.

Most teams build offline eval and call the job done. The teams that don't get burned are the ones running all three, with a standing discipline of adding to the regression set after every incident rather than treating the incident as a one-off.

Model selection as an architectural decision

A common anti-pattern once a team has working evals: pick one frontier model and run it for every task. It's expensive, slower than it needs to be, and above a certain volume stops being commercially viable at all.

Mature teams route tasks to model tiers instead of routing everything to the biggest model available:

The economics are the reason this is worth doing rather than a nice-to-have: in this course's own worked example, routing Sarah's triage system across six model tiers instead of running one frontier model everywhere cuts her monthly bill by 60 to 80 percent without reducing quality on any single task, because each task runs on a model sized for what it actually needs. Treat that range as a directional estimate from this specific worked system, not a guarantee for yours — the real number depends on your own task mix and current model choice.

Model selection, seen this way, is not a benchmark-chasing exercise. It's an architectural decision made per layer, measured per task, and revisited every quarter as new models release, which is a very different activity from picking "the best model" once and leaving it wired into everything.

Hardening keeps a system from being exploited. Evaluation tells you whether it's doing its job. Neither one, on its own, answers who in the organization is accountable for either, which is where the next lesson turns.