Give Your Ablation Recipe Arguments, Then Check Each Run
You keep typing the same experiment recipe: train with this config and seed, evaluate on the fixed 200-case subset, write the results with the commit hash. A reusable skill saves retyping it, and a repo whose experiments start from one tested recipe is easier for a colleague to rerun.
The skills documentation describes the controls. $ARGUMENTS is replaced by whatever follows the skill name, and $0, $1 (or $ARGUMENTS[0], $ARGUMENTS[1]) pick arguments by position. disable-model-invocation: true means only you can start the skill, which suits anything that spends GPU hours. It also means there is no trigger to test, unlike a skill the model picks from its description (see testing a Codex skill's trigger). The risk moves to the arguments.
---
name: run-ablation
description: Train and evaluate one ablation run
disable-model-invocation: true
---
Run `scripts/ablate.sh $0 $1` (the config path, then the seed).
Then report the path of the results file it wrote and the first five lines of it.Invoked as /run-ablation configs/small.yaml 3, Claude receives the recipe with the config and seed filled in. That makes the instruction reusable. It does not make the model's execution identical each time, and it does not prove every step ran. Keep the mechanical steps (training, evaluation, writing the file with git rev-parse HEAD) in scripts/ablate.sh, where they run the same way every time, and have the script fail loudly on a missing argument.
The check. After each run, confirm the results file records the config path and seed you passed and the current commit. Then test the failure case the documentation describes: an indexed placeholder with no argument at its position stays as literal text. Invoke /run-ablation configs/small.yaml with the seed missing. Claude will see a literal $1; your script should refuse to run rather than quietly training with a default seed.
The book's skills chapter shows how to test whether a skill's exact wording changes what the model does.
