Skills · Measure models
Statistics and experimental design
Taught here
Intervals, paired comparisons, sample sizes and controls: whether a difference is real.
Mentioned in 57 of 338 recorded research roles (17%) and 138 of 1,837 technical roles (8%), as of October 2, 2026.
Learn it here
- Book, Chapter 16: error bars and paired comparisons
- Did model A really beat model B?
- Design an experiment that measures the right thing
- Research Engineering and Evaluation Science (coming soon)
The course page has its planned syllabus and a sign-up, which tells us how many people want it.
These counts describe the October 2, 2026 snapshot; postings need a fresh check. Explore opportunities.
Everything here that teaches it
7 pieces. Pieces built around this skill come first in each list; the rest use it along the way.
Book chapters
-
Chapter 3: The agent loop: why reliability compounds, and a loop that stops
Derive why agent reliability compounds as p to the n, how recovery and retries change it, and measure both on a real model. Then build a bounded model and tool loop in Python.
-
Chapter 16: Evaluation as measurement: error bars and paired comparisons
Treat an evaluation as a measurement: Wilson intervals, standard errors clustered by case, McNemar's paired test, sample size and unbiased pass@k. Then build a harness that says what it checked.
Articles
-
Did Model A Really Beat Model B? Eval Variance, Measured
Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.
-
Calibrate an LLM Judge Before You Trust Its Score
Measured on Claude: asked for a one-word verdict, Haiku 4.5 called 74% of 500 math answers wrong when 52% were. A judge is two recalls, not one score. Derive them, their intervals and the correction, then play with them.
-
Measure Coding-Agent Uplift Honestly: Case Mix and Simpson's Paradox
A coding agent can look faster overall while being slower on every kind of task. Count the whole task, hold the case mix fixed, and see how Simpson's paradox produces a false speedup.
-
Read a Coding Benchmark Score: Contamination, Validity and Error Bars
A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.
Also in Measure models: Evals and benchmarks, Running experiments.
Who asks for it
- Research scientists and engineers 57 of 338
- Fellowships, residencies and internships 2 of 44
- Applied AI and forward-deployed 4 of 323
- ML systems and infrastructure 5 of 109
- Machine learning engineers 34 of 146
- Software engineering at labs 36 of 877
Recorded roles that mention it
138 recorded postings mention it. Here are the newest 60 research, fellowship, applied and ML-systems roles; the rest are on the role pages linked above. Each title opens the lab's own posting.
- Research Engineer, Mid-Training ↗
Cognition · Research scientists and engineers · PhD required
- Research Engineer, Post-Training ↗
Cognition · Research scientists and engineers · PhD required
- Partner Solutions Engineer ↗
Vercel (v0) · Applied AI and forward-deployed · No degree stated · $176k to $225k base
- Research Engineer, Asta ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD preferred · $129k to $193k base
- Research Engineer, Climate Modeling ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD preferred · $147k to $220k base
- Research Scientist, Climate Modeling ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD required · $167k to $261k base
- Senior Research Engineer, Olmo + Molmo ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD or equivalent experience · $174k to $261k base
- Research Engineer / Research Scientist, RL Frontiers ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Code RL (Reinforcement Learning) ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Post-Training Model Evaluations ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Pretraining Scaling ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $850k base
- Research Engineer, Pretraining Scaling - London ↗
Anthropic · Research scientists and engineers · No degree stated
- Staff Software Engineer, GTM AI Engineering ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $320k to $405k base
- Staff+ Software Engineer, ML Inference Path ↗
Anthropic · ML systems and infrastructure · No degree stated · $320k to $485k base
- Researcher, Evals ↗
Cartesia · Research scientists and engineers · No degree stated · $220k to $350k base
- Principal Research Engineer, Post-Training ↗
Character.AI · Research scientists and engineers · PhD or equivalent experience · $275k to $400k base
- Research Engineer, AI Safety & Alignment ↗
Character.AI · Research scientists and engineers · PhD or equivalent experience · $225k to $400k base
- Research Engineer, Post-Training (All Industry Levels) ↗
Character.AI · Research scientists and engineers · PhD or equivalent experience · $225k to $400k base
- Member of Technical Staff, Data Analysis and Evaluation ↗
Cohere · Research scientists and engineers · No degree stated
- Research Internship (Winter 2027) ↗
Cohere · Fellowships, residencies and internships · PhD required
- Senior Member of Technical Staff, Safety and Security for Agents ↗
Cohere · Research scientists and engineers · No degree stated
- Senior Research Engineer - Safety Tooling and Data ↗
Cohere · Research scientists and engineers · No degree stated
- Research Developer, Frontier AI Incubation, DeepMind ↗
Google DeepMind · Research scientists and engineers · Bachelor's or equivalent
- Research Engineer, Gemini Omni, DeepMind ↗
Google DeepMind · Research scientists and engineers · No degree stated
- Research Engineer, Generative Media, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD preferred
- Research Scientist, Gemini Safety and Behavior, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Safety Oversight, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Safety Oversight, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist (Applied LLMs), London ↗
Isomorphic Labs · Research scientists and engineers · PhD or equivalent experience
- Research Scientist (Machine Learning), London ↗
Isomorphic Labs · Research scientists and engineers · PhD or equivalent experience
- AI Research Assistant (PhD) ↗
Meta (Superintelligence Labs and FAIR) · Fellowships, residencies and internships · PhD required
- AI Research Scientist - Meta Superintelligence Labs (Technical Leadership) ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Research Engineer, Safety Evaluation ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Research Scientist Manager, AI for Wearables and Health ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · No degree stated
- Research Scientist, Polymer Physicist ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Research Scientist, World Modeling ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Staff Research Scientist, FAIR (RL / LLM's) ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD or equivalent experience
- AI Engineer, Product ↗
Mistral AI · Applied AI and forward-deployed · No degree stated
- Agent Post-Training Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Artifacts Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Computer Use Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Connectors Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Context Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Frontier Evals and Environments Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Applied AI Engineer, Quants ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Research Engineer, Codex ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Research Engineer, Frontier Evals & Environments ↗
OpenAI · Research scientists and engineers · No degree stated · $295k to $380k base
- Researcher, Synthetic RL ↗
OpenAI · Research scientists and engineers · No degree stated · $295k to $445k base
- Software Engineer, Monetization ML Infrastructure ↗
OpenAI · ML systems and infrastructure · No degree stated · $347k to $445k base
- System Performance Engineer, Consumer Devices ↗
OpenAI · ML systems and infrastructure · No degree stated · $347k to $385k base
- Computational Scientist, Structural & Thermal ↗
Periodic Labs · Research scientists and engineers · PhD or equivalent experience · $250k to $350k base
- Member of Technical Staff - Data Quality Engineer (Post-training) ↗
Reflection AI · Research scientists and engineers · No degree stated
- Member of Technical Staff - Data Quality Engineer (Pre-training) ↗
Reflection AI · Research scientists and engineers · No degree stated
- Member of Technical Staff - Evaluations ↗
Reflection AI · Research scientists and engineers · No degree stated
- Network Engineer, Supercomputing ↗
Thinking Machines Lab · ML systems and infrastructure · Bachelor's or equivalent · $350k to $475k base
- Reliability Engineer, Supercomputing ↗
Thinking Machines Lab · ML systems and infrastructure · Bachelor's or equivalent · $350k to $475k base
- Research Engineer, Infrastructure, Kernels ↗
Thinking Machines Lab · Research scientists and engineers · Bachelor's or equivalent · $350k to $475k base
- Research Engineer, Infrastructure, Numerics ↗
Thinking Machines Lab · Research scientists and engineers · Bachelor's or equivalent · $350k to $475k base
- Research Engineer, Infrastructure, Training Systems ↗
Thinking Machines Lab · Research scientists and engineers · Bachelor's or equivalent · $350k to $475k base
- Research Lead, Tinker, Fine-tuning Science ↗
Thinking Machines Lab · Research scientists and engineers · PhD or equivalent experience · $350k to $475k base
