Course in preparation · Coming soon
Research Engineering and Evaluation Science
Run experiments the way a lab team does: reproducibly, at scale, with evaluations you can defend. Design the evaluation, size it, run the sweep, and write up the result with its uncertainty.
Who it is for. Research-engineering and research-science candidates who need to show how they design, run and report experiments.
The planned syllabus
Planned lessons will help you understand each method, apply it to an AI problem, and judge whether the result holds up.
- 01
Reproducible experiments
Seeds, configurations and recorded model identifiers: an experiment someone else can rerun and get the same number.
- 02
Designing an evaluation
Build an evaluation set, split it, and check it for contamination before any system sees it.
- 03
Statistics for model comparisons
Paired bootstrap, variance across seeds and samples, and the sample size needed to detect a difference, derived and simulated.
- 04
Language-model judges
Measure a judge's agreement with human labels, its position and length biases, and how to calibrate it.
- 05
Scaling experiments
Fit a power law to a small sweep, and measure how far its extrapolation can be trusted.
- 06
Running experiments at scale
Sweeps, checkpoints, recovery from failure and cost accounting, so a long run survives the night.
- 07
Writing it up
The claim, the method, the uncertainty and the threats to validity, then defending them in an interview loop.
