U.S. teaching example: Amounts are fictional USD prices, stored as integer cents. This is a U.S. version of the shop exercise, not a currency conversion or a record of a real charge.
The same message, two correct answers
Consider this hypothetical evaluation task: "A customer wants six tubs of vanilla ice cream and four tubs of strawberry. Propose a purchase." The price list gives a total of 2600 cents, and the shop permits spending up to 3000 cents. Your evaluator accepts the agent's draft. Later, someone lowers the spending limit to 2000 cents and replays the same message. The draft total is still arithmetically correct, but the proposal now exceeds policy.
The evaluator needs both the request and the policy in force. Otherwise it can reject an agent that correctly refuses the purchase while accepting one that repeats the old, over-budget draft. This problem also occurs in ordinary software tests when a fixture omits configuration or external state. Agent evaluation makes those dependencies especially visible because tools can read several changing systems during one task.
Anthropic's engineering guide to agent evaluations distinguishes the task, the run's trajectory and the outcome. LangChain's guide to building environments and tasks describes tasks using an input, an environment and a test script, with environments defined separately for reuse. For this ordering exercise, those ideas give us a practical design: keep the stock, budget and available tools explicit, and attach an independently checked expected outcome to that exact combination.
Defining the agent evaluation environment
Before going further, a few terms need pinning down, because "environment" gets used loosely.
Input is the message or state the agent receives — the customer's request, in our example.
Environment is everything outside the agent's own reasoning that determines what a correct response would be: the current stock snapshot, the spending policy, the set of tools the agent is permitted to call, and any external state those tools read or write.
Oracle is the mechanism that judges whether the agent's output is correct. Crucially, the oracle has to be computed independently of the system under test. If the same arithmetic routine both generates the "expected" answer and checks the agent's answer, a shared bug in that routine will never be caught. This is the same discipline behind hand-calculating a baseline before running any code, which is exactly what a related teaching lab in this course does: it has the student compute six vanilla tubs at 250 cents plus four strawberry tubs at 275 cents on paper first, arriving at 2600 cents, before running anything (see Lab 2: Prompts, Roles, Tool Calls, and a Measured Harness). That lab treats the arithmetic helper used to build the fixture as insufficient evidence on its own — you need a hand-computed baseline as an independent check.
Permitted tools are the specific actions the agent is allowed to attempt, not a vague sense of "reasonable behavior." A tool boundary has to be enforced in code, separate from anything the prompt says.
Building the bundle: worked case
The table below is a hypothetical task specification. The two cases share quantities and prices; only the permitted spend changes. No purchasing service is called.
| Field | 3000 cents case | 2000 cents case |
|---|---|---|
| Input message | "Customer wants six vanilla, four strawberry" | Same message, verbatim |
| Environment: stock snapshot | Frozen: vanilla and strawberry both in stock, no shortage | Same snapshot |
| Environment: budget policy | 3000 cents spending limit | 2000 cents spending limit |
| Permitted tools | check_stock, propose_draft (no purchase) | Same tool set |
| Independently computed cost | 6×250 + 4×275 = 2600 cents | 2600 cents (arithmetic doesn't change) |
| Expected agent behavior | Accept: propose draft at 2600 cents | Refuse: 2600 cents exceeds 2000 cents limit |
| Oracle check | Requested quantities and total match; draft passes the 3000 cents policy | Draft is rejected for exceeding the 2000 cents policy |
| Side-effect boundary | No purchase tool is exposed; separately inspect attempted calls | Same prohibition and trace check |
Notice what stays fixed and what moves. The input message is identical in both columns. The independently computed cost is identical too, because arithmetic doesn't care about policy. What changes is the environment's budget field, and that single change flips the expected outcome from "accept" to "refuse." A harness can include the environment in a composite input, or store it separately and refer to its version. Either representation works if the expected result is tied to the exact environment. An input message alone leaves the budget implicit and makes accidental reuse much easier.
A proposal that passes the 3000 cents case does not remain valid when replayed under the 2000 cents case with the same "expected answer" copied over. The environment is part of the task's contract, not an incidental detail you can vary for free.
Hiding the oracle from the generator
A second failure mode is letting the agent, or the code path that builds the agent's fixtures, see the expected answer. If the environment definition includes a field like expected_total: 2600 and that field is readable by the same process constructing the prompt, you have built a system that can pass by copying rather than by reasoning. The fix is structural, not cosmetic: the oracle's expected value lives in a separate object that the agent's context never touches, and the harness that scores the run reads from that object only after the agent has committed to a proposal.
Inspect the exact serialized request and every tool response the agent can receive. Searching for private oracle field names catches direct leaks, but absence of those names does not prove isolation: a value can be copied under another name or appear in a retrieved document. Build the agent payload from an allowlist of task fields, keep expected outcomes in a separate evaluator input, and inspect the resulting bytes. The price list legitimately lets the agent calculate 2600 cents; the prohibited shortcut is giving it the evaluator's answer record.
This data-flow view helps review that boundary. The agent receives task facts and tools; the grader receives the committed proposal plus the private oracle. An arrow from the oracle into the agent payload would reveal a leak before you run the test.
Ordered steps for freezing a task
Write the input message and freeze it exactly as the agent will receive it, including any system-level framing.
Define the environment separately: stock snapshot, budget policy, and the exact list of permitted tools, each as its own versioned field, none of them containing the expected result.
Compute the expected result independently, by hand or with a routine that shares no code with whatever builds the agent's prompt or tools.
Change one environment variable at a time when testing variants, and re-derive the expected result each time rather than copying the previous one forward.
That fourth step is where most reused test suites go wrong. It's tempting, when you already have a passing 3000 cents case, to just drop the budget number to 2000 cents and assume the rest of the bundle still applies. It does not. The input and permitted tools can stay the same; the expected outcome cannot be assumed to stay the same, because the environment moved.
Even when your oracle correctly checks that a draft's numbers are right, that check says nothing about whether any real transaction occurred. Keep "the proposal matches the computed total" and "a purchase happened" as separately tracked claims. Conflating them is a different bug from the environment-reuse bug, but it shows up in the same kind of harness.
Checkpoint
Check the decision
A changed environment can preserve the answer
Changing the budget from 3000 cents to 4000 cents leaves this 2600 cents draft valid. That is a useful control case: a new environment version requires reviewing the expected result, but does not necessarily require a different answer. Lowering it to 2000 cents does. Include both cases so your harness tests the policy boundary rather than treating every version change as an automatic rejection.
| Budget | Total | Expected draft decision |
|---|---|---|
| 2000 cents | 2600 cents | Reject as over budget |
| 2600 cents | 2600 cents | Accept under an inclusive spending limit |
| 3000 cents | 2600 cents | Accept |
| 4000 cents | 2600 cents | Accept |
The equality case exposes another hidden assumption: does "up to" include the limit? Here it does. Write that rule explicitly. A production workflow might also reserve funds, apply tax or check stock concurrently; those would be additional task facts and constraints. They should not appear only after a failed deployment.
An environment field may affect timing, permissions or side effects even when the final text stays unchanged. Inspect those outcomes before deciding a field is irrelevant. For example, removing a purchase tool should leave the drafted quantities unchanged while making execution unavailable. A text-only grader would miss that difference.
What the course lab contributes
Lab 2 in the ITAM course supplies the related ice-cream arithmetic, a validator and a bounded retry exercise. Its written exercise is a source for the teaching method, not a receipt that this article's proposed environment variants have run. The tables here specify expected behavior; an implementation still needs to record its actual draft decisions and tool attempts against them.
Next action
Take one existing test you already trust, even a simple one, and write down its environment fields explicitly, separate from its input and expected output. Then ask whether changing one of those fields would force the expected output to change. If the answer is yes for any field, that field was already part of the contract; you were just storing it implicitly, which is the condition that let the 3000 cents case get replayed as if it still meant something at 2000 cents.
If this worked case made sense, the full lab builds the validator, retry budget and tool-boundary checks behind it step by step.
