Move to the Agent SDK When You Need to Own the Run
Running an agent by hand from a terminal is how you try an idea. Running it over 200 eval tasks every night is an experiment, and an experiment needs what a terminal session does not give you: a transcript per task, a clean way to cancel, a record of cost, and permissions you set in code rather than click through.
That is the point where the Claude Agent SDK is worth evaluating. Anthropic's overview says it gives you "the same tools, agent loop, and context management that power Claude Code, programmable in Python and TypeScript." The same page lists the neighbors. From another language, run the CLI as a subprocess with -p and --output-format json. To write the tool loop yourself, call the Claude API through the Client SDK.
The choice shapes what you can report. Agent success compounds across steps. If each of 20 tool steps succeeds independently with probability 0.98, the whole task succeeds with probability 0.98^20 ≈ 0.67. You can only find the weak step if your runtime records every step. The agent loop derives this and builds a loop that stops.
The SDK does not set your policy. You still decide which files and tools each run may touch, which actions need approval, and where transcripts are kept.
The check. Pick 10 tasks from your eval set. Run them once through claude -p --output-format json in a shell loop and once through a minimal SDK script. For each path, answer three questions from the saved output alone:
- Which tasks passed?
- In the ones that failed, which step failed?
- What state did a run you canceled halfway leave behind?
The path where you can answer all three without rerunning anything is the one your experiment can rely on. If neither can, that is your first engineering task, before any model comparison.
