First Contact: A Lazy Brief, and the One That Isn't
Xolo Consulting, a small fictional bookkeeping client built for this course, gets its first real visitor. This unit walks through two attempts at the exact same task, reconciling a month of invoices against bank deposits, run against the same data, with the same model, minutes apart. The reference manual is behind you now. The first attempt is lazy on purpose, eight words, no more, and it fails in a way that turns out to be instructive once you audit it against the seven traps: seven specific, deliberately seeded errors in Xolo's books, from a misspelled client name to a duplicate deposit, that every reconciliation run in this course gets graded against. The second attempt is the same task, briefed properly, run in plan mode, and it's the first time this class the scoreboard actually moves.
This is the second unit of Class 3. It assumes you've read the reference manual in the previous unit and have Claude Code running against your own clone of the Xolo template. If you haven't installed it yet, do it now; the two demos in this unit are the two-minute job the previous unit promised.
Orienting a fresh session
The very first thing worth watching, before any task is given at all, is what happens the moment Claude Code starts inside the Xolo folder and gets asked a single orienting question: what is this project, and what rules do you operate under?
It recites CLAUDE.md, Xolo's project memory file, without being told anything about it in the conversation. Nobody pasted the file's contents into the chat. Nobody said "read your instructions first." The model simply already knows the map of the repository, the invariant that data files never get modified, the rule about matching records on IDs rather than names, because that file gets read into every session's context automatically, the moment the session starts. Where did that knowledge come from, if nobody typed it?
/context answers the question directly: it shows a breakdown of exactly what's currently occupying the window, and CLAUDE.md's contents sit right there in the pie, alongside the system prompt and the tool definitions. This is term four of last class's equation, itemized in front of you rather than left as an abstraction. A quick pass through /status confirms the same picture from a different angle: which model is active, which memory files are loaded, whether any MCP servers are connected. None are, yet.
One ordinary permission prompt is worth walking through here too, because it's the first one you'll actually see fire. Asking the session to count rows in the CRM invoice file triggers a Bash call running a small python one-liner, and the prompt appears exactly as described in the previous unit: the literal command, three numbered choices. Answering "allow once" lets it run. Asking for a second row count on a different file triggers the same prompt again, and this time the "always allow for python commands" option is the one worth taking, because it's the concrete moment a session gets smoother by approving a pattern instead of a single instance.
The experiment: what happens without the file
Here's where the unit turns into something worth actually sitting with. Somewhere on your machine, in a second terminal, rename CLAUDE.md away, mv CLAUDE.md _CLAUDE.md, so the project temporarily has no memory file at all. Run /clear to wipe the session's context clean. Then give it the laziest brief that could plausibly describe the task: "reconcile the invoices against the bank deposits." Eight words.
Without CLAUDE.md's join rule to constrain it, a naive run typically matches invoices to deposits by client name, because names are how humans normally talk about clients, and nothing in an eight-word instruction says otherwise. That's exactly where trap zero, the client-name-spelling bait seeded across Xolo's CRM and bank records, bites. The run answers inline in the chat rather than writing a report anyone can hand off. It finds some of the missing invoices, because a few of them are obvious even through a bad join. It misses the duplicate deposit and the currency mismatch entirely, because nothing asked it to check for either. And there's no control total anywhere in the output, no number proving the gap between the two files has been fully explained, which means there's no way to know, from the output alone, what else it might have missed.
Audit the naive run against the trap board, one trap at a time, the same way you'll audit every run for the rest of the class. Out of the seven seeded problems, a typical naive run catches something like three or four, and the deeper issue isn't the raw count, it's that there's no way to know from the output itself which of the three remaining traps were missed versus which were checked and correctly ruled out. An unverifiable partial answer is worse than it looks, because it invites false confidence.
Put the memory file back, mv _CLAUDE.md CLAUDE.md, before moving on. The point of the experiment isn't that the file is magic. It's that the same model, on the same data, produces a dramatically weaker result the moment its instructions get thinner, which sets up the actual diagnosis in the next section. Right now the scoreboard reads something like three out of seven, and unverifiably so, which for the purposes of this class's honest accounting rounds down to not really moving at all.
The autopsy: why eight words failed
Nothing was wrong with the model in that naive run. It received eight words and a folder full of files, and it filled every gap left open with its own statistical prior about what "reconcile the invoices" usually means online. Of course it joined on names; names are how humans normally talk about clients in conversation. Of course it answered inline in the chat instead of writing a structured file; that's the shape a typical online exchange about "reconciling" something takes. The failure was upstream of the model entirely. It was in the brief.
The five-part brief, annotated
Here's the shape worth stealing, the exact text of a properly constructed brief for this same reconciliation task, annotated part by part:
Read it once as a prompt. Then read it again as what it actually is: a work order you could hand to a competent human and expect a professional result back. Every part earns its place, and none of them are folklore.
Naming the exact files up front saves the exploration turns the agent would otherwise spend hunting for them, and it saves the window space those exploration turns would burn along the way.
"Reconcile" alone means nothing specific. Naming the exact file, the exact table shape, and the requirement of one row per exception replaces the model's generic prior of "write a report" with your own concrete acceptance criterion.
Rules prune the search space before the model ever starts guessing. One line about matching on IDs rather than names is enough to kill trap zero outright, the exact trap that ate the naive run a few minutes ago.
This is the part that does more than it looks like it does. Demanding control totals and a fully-explained gap doesn't just give you something to check afterward. It changes what the model optimizes for during the run itself, because producing that evidence is now part of the task, not an afterthought bolted onto the end.
Requiring a plan before execution is the plan-mode gate from the previous unit, applied here directly. A misunderstanding caught in the plan costs seconds. The same misunderstanding, caught after a wrong report has already been written, costs the twenty minutes it takes to notice something's off and redo the whole thing.
A folklore check
Before watching the properly briefed run, a short detour is worth taking, because the internet is full of prompt-engineering advice, and some of it survives contact with how this loop actually works while a good deal of it doesn't.
Show, don't describe: one concrete example of the output you want beats three paragraphs of adjectives describing it, and it's the cheapest specification language that exists. Paste the real thing: an actual error message, actual sample rows, an actual screenshot pasted with Ctrl+V, beat a description of any of those every time. Decompose: one bounded task per session, chained across sessions rather than piled into one, protects the context window from the exact kind of degradation /context exists to diagnose. And fix the brief, not the output: Esc Esc, edit the instruction, rerun, the same habit named in the previous unit, because instruction edits compound across a session while output edits evaporate the moment you /clear.
Telling the model it's "the world's best analyst" adds tokens and changes nothing measurable about its behavior. Pasting everything you have "just in case" degrades the exact window it's trying to fill, the opposite of what it's meant to do. Asking "are you sure?" fails structurally: the same context that produced an error is the context being asked to re-grade that error, which is asking a student to re-mark their own exam with the same misunderstanding still in their head. The fix for that one isn't a better phrasing, it's an independent context altogether, and that idea gets a proper name two lessons from now. And prompt-tweaking a task whose scope is simply too large never works, in any phrasing, because the problem was never the wording. It was the scope.
:::checkpoint{id="itam-w03s2-are-you-sure" question="Why does asking a model "are you sure?" fail to reliably catch its own mistakes, when the fix isn't a better way to phrase the question?" options="The model is not trained to answer follow-up questions,The same context that produced the original error is the context re-grading it, so the same blind spot is likely to persist,Asking follow-up questions uses too many tokens to be worth it,It only fails on reconciliation tasks specifically" answer="2"} "Are you sure?" asks the model to check its own work using the exact same reasoning, the same assumptions, and the same blind spots that produced the error in the first place. It's not a wording problem, so no rephrasing fixes it. What actually works is verification from a separate context with no visibility into how the original answer was produced, which can't inherit the same mistake because it never saw the reasoning that led to it. :::
The properly briefed run
Now the same task, run for real, in plan mode.
/clear first, to wipe the naive attempt out of context entirely, then Shift+Tab into plan mode, watch the status line confirm it, and paste the five-part brief above.
The plan that comes back explores the two data files, proposes a regex for pulling the invoice ID out of the bank concept field, lays out the exception buckets, and states it will include control totals, all without having touched a single file yet. Worth noticing specifically: it caught the name-spelling hazard on its own, from the one line in the brief about matching on IDs rather than names, without that hazard ever being pointed at directly.
Before approving the plan, one more instruction lands: "Also break the exceptions down by client, and flag any invoice past its client's payment terms." That's the ten-second correction from the previous unit's arithmetic, made real. Nothing has been built yet, so nothing needs to be unwound or redone. The plan simply grows to include the new requirement, and execution starts from the corrected version instead of a patched-after-the-fact one.
Approving the plan moves the session into execution, and the permission prompts along the way are worth narrating as they happen. Running python is pre-approved from the earlier session, from taking the "always allow" option, so no prompt interrupts that part. Writing the actual report file to reports/ does prompt, because writing to a new path is exactly the kind of consequential action the gate exists to catch. Partway through, a TodoWrite call may appear on its own, a visible checklist for the multi-step task, the organize family showing up unprompted, exactly as described in the tool belt from the previous unit.
The audit, and the scoreboard moves
Now the real payoff: auditing the properly briefed report against the same trap board used on the naive run, trap by trap.
| Trap | Naive run | Properly briefed run |
|---|---|---|
| Three uncollected invoices | Partial, unverifiable | Found, with control totals closing the gap to the cent |
| Transposed-digit amount mismatch | Missed | Found, delta shown |
| USD deposit, currency mismatch | Missed | Found |
| Duplicate deposit | Missed | Found |
| Client-name spelling bait | Fell for it | Avoided, caught in the plan itself |
| Self-contradicting amendment | Never checked | Still 0, no procedure exists for contracts yet |
| Unreadable scanned contract | Never checked | Still 0, same reason |
Five of seven, and evidenced this time, not guessed at. The control totals in the report close the gap between the two files down to the cent, which means there's no silent remainder hiding a sixth or seventh problem the report simply didn't mention. The two contract traps stay at zero, honestly, because nothing in this brief asked the agent to look at contracts at all, and that's not a failure of this run, it's an accurate description of its scope. /cost for the session comes back at a few cents, for what would be an analyst's afternoon of work, verified in about two minutes by reading the evidence section rather than trusting the summary at the top.
The turn
Sit with what just happened for a second. That five-part brief, the one that actually worked, that caught the name hazard on its own, that closed the gap to the cent, lives entirely inside a chat window that dies the moment someone runs /clear. Next month, whoever's running Xolo's books retypes this same reconciliation from memory, forgets the exact join rule, and trap zero eats the report all over again.
Everything of real value that happened in this unit is currently trapped in the most perishable container computing has ever produced: a conversation. The next lesson is about the one move that fixes this, applied five separate ways, and it starts from nine words worth remembering for the rest of this course and the rest of your career with this tool: the conversation is volatile. The files are permanent.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
