GuideBuilding & Deploying AI
Agent Evaluation Environment: Freeze the Task Contract
An agent evaluation environment must freeze inputs, tools and an independent oracle, or the same message silently grades against two different answers.
10 min read
Subject
Filed under Building & Deploying AI
Evaluation, permissions, customer delivery, rollout, operation and handover
What this shelf holds
What it takes to put an AI system in front of a customer and keep it there: evaluation you can trust, permissions that mean something, retries that do not repeat a payment, and a handover the receiving team can verify.
These are engineering pieces. Most work through one decision with a small, inspectable example, and say what the example does and does not prove. This is where the site's Harness Engineering writing lives.
Picked by the editor
Start with these
GuideBuilding & Deploying AI
Agent Regression Dataset: Save the Failure, Not the Fix
Learn to turn one failed agent run into an agent regression dataset that tests recovery without pretending to know the original cause.
12 min read
GuideBuilding & Deploying AI
AI Agent Per User Credentials: Who Owns the Key?
AI agent per user credentials means separating who invokes an agent from which account its tools use, so one caller never inherits another's access.
11 min read
GuideBuilding & Deploying AI
Agent Tool Idempotency: Reconcile an Uncertain Write
A timed-out tool call leaves you unsure a write happened. Choose an idempotency key, and reconcile the result before retrying anything.
12 min read
GuideBuilding & Deploying AI
Stateless MCP Application State: Where Identity Really Lives
Keep MCP job state durable while connections change. Check authenticated identity and job permissions on each protected request before resuming work.
10 min readUpdated
GuideBuilding & Deploying AI
Write an AI Agent Handoff Checklist That Actually Verifies
Learn to write an AI agent handoff checklist that lets a new engineer verify state and evidence instead of trusting a summary.
13 min read
Courses and chapters
Learn it properly
- Why Agents FailWhy agent projects fail in production despite working demos.
- Two ArchitecturesPlace an agent project on the two-axis grid before you pick a stack.
- Tools, Memory and Multi-Agent SystemsDesign tools an agent can call safely and memory it can trust.
- Build a Dark Software FactoryFive free lessons on testing, release and recovery with coding agents.
- The bookBuild a Python agent with tools, memory and recovery, chapter by chapter.
Newest on this shelf
Most recent
GuideBuilding & Deploying AI
AI Approval Stale Data: Why Snapshots Expire
A human approval binds to one data snapshot. Learn why AI approval stale data checks must recheck inputs before acting, not just before asking.
12 min read
NoteBuilding & Deploying AI
Ask for the Smallest Useful Model Output
If your application needs a category and one reason, a page of explanation creates extra reading and processing work. Define the output your next step can actually use. For a document-routing exercise, ask for an allowed category, a short supporting reason…
openailatency
GuideBuilding & Deploying AI
AI Tool Error Handling: Don't Fake an Empty Result
AI tool error handling must distinguish an empty catalog from unavailable data. Use separate result types and a mutation test that catches a swallowed timeout.
11 min read
NoteBuilding & Deploying AI
Measure Where the Waiting Time Actually Goes
The model may not be the slowest part of your AI workflow. Retrieval, tool calls and repeated validation attempts can consume most of the waiting time. For a synthetic support task, record the duration of input preparation, the model request, each tool call…
openailatency
GuideBuilding & Deploying AI
AI Agent Tool Design: One Job Per Tool Boundary
AI agent tool design should expose effects and permission boundaries. Compare an overloaded inventory function with explicit read and draft tools.
13 min read
NoteBuilding & Deploying AI
Use Batch Work Only When the Deadline Fits
An overnight document-classification job and a live customer conversation have different timing needs. Choose the API workflow around the deadline. For work that can wait, evaluate the Batch API using its current completion-window and pricing documentation…
openaibatch-processing
