Skills · Measure models
Statistics and experimental design
Taught here
Intervals, paired comparisons, sample sizes and controls: whether a difference is real.
Mentioned in 58 of 339 recorded research roles (17%) and 140 of 1,827 technical roles (8%), in this recorded board.
Learn it here
- Book, Chapter 16: error bars and paired comparisons
- Did model A really beat model B?
- Design an experiment that measures the right thing
- Research Engineering and Evaluation Science (coming soon)
The course page has its planned syllabus and a sign-up, which tells us how many people want it.
Everything here that teaches it
9 pieces. Pieces built around this skill come first in each list; the rest use it along the way.
Book chapters
-
Chapter 3: The agent loop: why reliability compounds, and a loop that stops
Derive why agent reliability compounds as p to the n, how recovery and retries change it, and measure both on a real model. Then build a bounded model and tool loop in Python.
-
Chapter 16: Evaluation as measurement: error bars and paired comparisons
Treat an evaluation as a measurement: Wilson intervals, standard errors clustered by case, McNemar's paired test, sample size and unbiased pass@k. Then build a harness that says what it checked.
Articles
-
Write the Acceptance Spec Before You Build an Eval Judge
An LLM judge can only grade against a written, versioned policy. Name the owner, the rubric, the labeled set and the pass rule before you build it, and test the rubric on people first.
-
Did Model A Really Beat Model B? Eval Variance, Measured
Measured on 100 AIME problems, ten runs each: single runs of GPT-6 Luna scored anywhere from 88% to 98%. Derive task-level errors, pass@k and pass^k, and paired intervals, then play with them.
-
Calibrate an LLM Judge Before You Trust Its Score
Measured on Claude: asked for a one-word verdict, Haiku 4.5 called 74% of 500 math answers wrong when 52% were. A judge is two recalls, not one score. Derive them, their intervals and the correction, then play with them.
-
Measure Coding-Agent Uplift Honestly: Case Mix and Simpson's Paradox
A coding agent can look faster overall while being slower on every kind of task. Count the whole task, hold the case mix fixed, and see how Simpson's paradox produces a false speedup.
-
Read a Coding Benchmark Score: Contamination, Validity and Error Bars
A benchmark score is an estimate from a sample of tasks, graded by an oracle that can be wrong, on data a model may have seen. Put error bars on it and test contamination and oracle validity.
Notes
-
Measure Human Agreement Before You Trust an LLM Judge
An LLM judge cannot give a stable quality signal if the people who wrote the rubric disagree about which model outputs should pass.
-
Test a Model on the Hard Slice, and Put an Interval on It
An eval average can look healthy while the model fails the cases that decide whether it is fit for the task.
Also in Measure models: Evals and benchmarks, Running experiments.
Who asks for it
- Research scientists and engineers 58 of 339
- Fellowships, residencies and internships 1 of 32
- Applied AI and forward-deployed 4 of 326
- ML systems and infrastructure 6 of 102
- Machine learning engineers 33 of 143
- Software engineering at labs 38 of 885
Recorded roles that mention it
140 recorded postings mention it. Here are the newest 60 research, fellowship, applied and ML-systems roles; the rest are on the role pages linked above. Each title opens the lab's own posting.
- Staff Research Engineer, Multi-Agent Scaling ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer - Eval Platform ↗
Mistral AI · Research scientists and engineers · Master's, PhD or equivalent
- Member of Technical Staff, ML Platform ↗
Runway · ML systems and infrastructure · No degree stated · $240k to $290k base
- Research Engineer, Mid-Training ↗
Cognition · Research scientists and engineers · PhD required
- Research Engineer, Post-Training ↗
Cognition · Research scientists and engineers · PhD required
- Partner Solutions Engineer ↗
Vercel (v0) · Applied AI and forward-deployed · No degree stated · $176k to $225k base
- Research Engineer, Climate Modeling ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD preferred · $147k to $220k base
- Research Scientist, Climate Modeling ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD required · $167k to $261k base
- Senior Research Engineer, Olmo + Molmo ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD or equivalent experience · $174k to $261k base
- Research Engineer / Research Scientist, RL Frontiers ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Code RL (Reinforcement Learning) ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Post-Training Model Evaluations ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Pretraining Scaling ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $850k base
- Research Engineer, Pretraining Scaling - London ↗
Anthropic · Research scientists and engineers · No degree stated
- Staff Software Engineer, GTM AI Engineering ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $320k to $405k base
- Staff+ Software Engineer, ML Inference Path ↗
Anthropic · ML systems and infrastructure · No degree stated · $320k to $485k base
- Researcher, Evals ↗
Cartesia · Research scientists and engineers · No degree stated · $220k to $350k base
- Principal Research Engineer, Post-Training ↗
Character.AI · Research scientists and engineers · PhD or equivalent experience · $275k to $400k base
- Research Engineer, AI Safety & Alignment ↗
Character.AI · Research scientists and engineers · PhD or equivalent experience · $225k to $400k base
- Research Engineer, Post-Training (All Industry Levels) ↗
Character.AI · Research scientists and engineers · PhD or equivalent experience · $225k to $400k base
- Member of Technical Staff, Data Analysis and Evaluation ↗
Cohere · Research scientists and engineers · No degree stated
- Research Internship (Winter 2027) ↗
Cohere · Fellowships, residencies and internships · PhD required
- Senior Member of Technical Staff, Safety and Security for Agents ↗
Cohere · Research scientists and engineers · No degree stated
- Senior Research Engineer - Safety Tooling and Data ↗
Cohere · Research scientists and engineers · No degree stated
- Research Developer, Frontier AI Incubation, DeepMind ↗
Google DeepMind · Research scientists and engineers · Bachelor's or equivalent
- Research Engineer, Gemini Omni, DeepMind ↗
Google DeepMind · Research scientists and engineers · No degree stated
- Research Engineer, Generative Media, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD preferred
- Research Scientist, Gemini Safety and Behavior, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Safety Oversight, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Safety Oversight, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist (Applied LLMs), London ↗
Isomorphic Labs · Research scientists and engineers · PhD or equivalent experience
- Research Scientist (Machine Learning), London ↗
Isomorphic Labs · Research scientists and engineers · PhD or equivalent experience
- AI Research Scientist - Meta Superintelligence Labs (Technical Leadership) ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Research Engineer, Safety Evaluation ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Research Scientist Manager, AI for Wearables and Health ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · No degree stated
- Research Scientist, Polymer Physicist ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Research Scientist, World Modeling ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD required
- Staff Research Scientist, FAIR (RL / LLM's) ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD or equivalent experience
- AI Engineer, Product ↗
Mistral AI · Applied AI and forward-deployed · No degree stated
- Agent Post-Training Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Artifacts Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Computer Use Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Connectors Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Context Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Frontier Evals and Environments Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Applied AI Engineer, Quants ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Research Engineer, Codex ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Research Engineer, Frontier Evals & Environments ↗
OpenAI · Research scientists and engineers · No degree stated · $295k to $380k base
- Researcher, Synthetic RL ↗
OpenAI · Research scientists and engineers · No degree stated · $295k to $445k base
- Software Engineer, Monetization ML Infrastructure ↗
OpenAI · ML systems and infrastructure · No degree stated · $347k to $445k base
- System Performance Engineer, Consumer Devices ↗
OpenAI · ML systems and infrastructure · No degree stated · $347k to $385k base
- Computational Scientist, Structural & Thermal ↗
Periodic Labs · Research scientists and engineers · PhD or equivalent experience · $250k to $350k base
- Member of Technical Staff - Data Quality Engineer (Post-training) ↗
Reflection AI · Research scientists and engineers · No degree stated
- Member of Technical Staff - Data Quality Engineer (Pre-training) ↗
Reflection AI · Research scientists and engineers · No degree stated
- Member of Technical Staff - Evaluations ↗
Reflection AI · Research scientists and engineers · No degree stated
- Network Engineer, Supercomputing ↗
Thinking Machines Lab · ML systems and infrastructure · Bachelor's or equivalent · $350k to $475k base
- Reliability Engineer, Supercomputing ↗
Thinking Machines Lab · ML systems and infrastructure · Bachelor's or equivalent · $350k to $475k base
- Research Engineer, Infrastructure, Kernels ↗
Thinking Machines Lab · Research scientists and engineers · Bachelor's or equivalent · $350k to $475k base
- Research Engineer, Infrastructure, Numerics ↗
Thinking Machines Lab · Research scientists and engineers · Bachelor's or equivalent · $350k to $475k base
- Research Engineer, Infrastructure, Training Systems ↗
Thinking Machines Lab · Research scientists and engineers · Bachelor's or equivalent · $350k to $475k base
