All sections
Latest
Everything published, newest first.
The log
Entries 101–120 of 167 · by original publication date
3 entries
Why Schema-Valid Model Output Can Still Be Wrong, and How to Check It
A model's eval plan can pass every schema check and still use the test split, drop cases or overspend. Check it against trusted config, and measure how often valid means wrong.11 min readstructured-outputevaluation
Write a Handoff the Next Engineer or Agent Can Verify
A handoff for a half-finished sweep or eval should let the receiver check every claim, not trust it. Attach a check to each claim, separate claimed sends from receipts, and see why spot-checking is not enough.9 min readagent handoffresearch engineering
1 entry
Give a Reasoning Model the Budget and the Success Criteria
OpenAI's reasoning guide gives short advice. Keep prompts simple and direct. Skip "think step by step," because these models reason internally.Quick thoughtopenaiprompting1 entry
Record the Model Snapshot in Every Row of Your Results Table
A saved prompt is not an experiment record. The model snapshot, the sampling settings and the eval set can each move the score, and a results table that…Quick thoughtopenaievaluation1 entry
Freeze the Eval Before You Tune the Prompt, Then Hold Some Out
If you decide what "better" means after reading each new answer, prompt tuning will always look like progress. Freeze the evaluation first.Quick thoughtopenaievaluation1 entry
A Valid JSON Object Can Still Hold the Wrong Number
OpenAI's Structured Outputs guarantees that the model's response adheres to the JSON Schema you supply.Quick thoughtopenaistructured-output1 entry
Test a Model on the Hard Slice, and Put an Interval on It
An eval average can look healthy while the model fails the cases that decide whether it is fit for the task.Quick thoughtopenaievaluation1 entry
Ask Codex to Trace a Generated Results Table to Its Source
The results table in your paper repo shows 71.4 where the logs say 74.1. Editing the table by hand fixes the PDF until the next regeneration, and until then…Quick thoughtcodexcode-review1 entry
Turn a Repeated Results Check Into a Tested Codex Skill
Every time a sweep finishes, you paste the same instructions: check that each results file has the required fields, that no item id appears twice, and that…Quick thoughtcodexskills1 entry
Gate a Codex Step on Its Status, Its Schema and Its Numbers
A pipeline of five steps that each pass a bad artifact along 5% of the time delivers a clean result only 0.95⁵ ≈ 0.77 of the time.Quick thoughtcodexautomation1 entry
Pilot a Codex Migration on Two Files, Then Diff the Requests
You need to move 40 experiment configs to a new format: a renamed field, a new default, a changed prompt-template path.Quick thoughtcodextesting1 entry
Give a Codex Skill One Request It Must Refuse: Publishing
A skill that builds a results table from eval logs looks excellent on the request it was written for.Quick thoughtcodexskills1 entry
Give Codex Write Access to the Repo Without Giving It the Network
Fixing a bug in an eval script needs write access to the repository. It does not need the network. But if the script's tests call a paid model API, or…Quick thoughtcodexpermissions1 entry
Write Review Rules Around Your Experiment's Invariants
"Review carefully" gives a code reviewer nothing to prioritize. An invariant does. In research code, the invariants are the rules that keep a number honest: -…Quick thoughtcodexcode-review1 entry
Pin the Working Directory of a Scheduled Codex Run
A reviewer asks where a number came from. The answer you want is a commit and a command. A nightly eval that runs Codex from cron or CI starts wherever the…Quick thoughtcodexautomationUpdated1 entry
Parse Codex's Final Result, Not Its Progress Stream
A sweep driver calls codex exec once per config and builds a results table. If it parses everything the command prints, two things go wrong.Quick thoughtcodexautomation1 entry
A Codex Worktree's Result Depends on Files Git Doesn't Carry
Codex ran your ablation in a separate worktree and reports a two-point gain. Before that number goes in a results table, to a teammate or into the trash with…Quick thoughtcodexworktrees1 entry
Check Which AGENTS.md Files Codex Loaded Before Adding Rules
Before a model can follow a rule, the rule has to be in its context. Chapter 21 treats that context as a budget assembled from parts; for Codex, the…Quick thoughtcodexinstructionsUpdated1 entry
Find the Change You Never Asked the Agent to Make
You asked an agent to fix an axis label on a results plot. The diff also switched the F1 score from macro to micro averaging, because the agent read it as a…Quick thoughtclaude-codecode-reviewContinued on the next page.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
