Turn a Repeated Results Check Into a Tested Codex Skill
Every time a sweep finishes, you paste the same instructions: check that each results file has the required fields, that no item id appears twice, and that the row count matches the eval set. The reusable asset is that procedure and its test cases, not the chat history.
OpenAI's Codex documentation defines a skill as "a directory with a SKILL.md file plus optional scripts and references," and SKILL.md must include name and description. Repository skills live in .agents/skills. The skills guide describes two ways in. You can invoke a skill explicitly by typing $skill-name, or Codex invokes it implicitly when your task matches its description. At first Codex loads only each skill's name, description and path; the full SKILL.md loads only when the skill is chosen. So the description is behavior, and behavior can be tested.
Write the skill for one job and say where it stops. The guide asks for "concise descriptions with clear scope and boundaries," with the key use case and trigger words up front:
name: check-results
description: Check an eval results JSONL against the eval manifest and report
missing fields, duplicate item ids and count mismatches. Not for plots,
training logs or fixing the file.Leave related jobs out. A validator that starts deleting duplicate rows has changed your data, and your results table with it, without anyone deciding that it should.
Test what it does. Keep two fixtures beside the skill: a valid results file and a broken one with one missing field and one duplicated id. After every edit to the instructions, run the skill on both. Expected: no findings on the first, exactly two on the second.
Test when it loads. This is the part that fails silently. Write 10 prompts that should trigger the skill ("check the results from last night's sweep") and 10 near misses that should not ("summarize the training log", "plot the loss curve"). Run each in a fresh session and record whether the skill was chosen. Report two numbers: misses, where it should have fired and did not, and false fires. A miss needs trigger words; a false fire needs a sharper boundary.
Ten prompts a side only reveal gross problems. With zero false fires in n prompts, the 95% upper bound on the false-fire rate is 1 − 0.05^(1/n), about 26% at n = 10. Treat even one false fire as a reason to tighten the description, and add prompts before you trust a clean score.
Keep the description short for one more reason. The guide caps the initial list of skills at 2% of the model's context window, or 8,000 characters when the window size is unknown. At about 400 characters per description, that holds about 20 skills at full length. Front-loaded trigger words are what still match when descriptions get shortened.
A skill that loads correctly can still act outside its job when asked to. Give a Codex skill one request it must refuse tests that boundary by reading which commands ran. Chapter 7 measures how much a prompt's exact wording moves a model, which is why a skill's words need testing rather than review by eye.
