Skills · Measure models
Running experiments
Partly taught here
Turn a vague behavior into a hypothesis, a pipeline and a result: ablations, baselines and research taste.
Mentioned in 68 of 339 recorded research roles (20%) and 162 of 1,827 technical roles (9%), in this recorded board.
Learn it here
- Find the failing stage before you change the prompt
- Book, Chapter 17: optimizing against an evaluation
- Research Engineering and Evaluation Science (coming soon)
The course page has its planned syllabus and a sign-up, which tells us how many people want it.
Everything here that teaches it
7 pieces. Pieces built around this skill come first in each list; the rest use it along the way.
Articles
-
Design a Prompt-Caching Experiment That Measures the Right Thing
Derive when prompt caching pays from Anthropic's write premium and read discount, then run a cold, warm and changed-prefix protocol that keeps cost, context and latency apart.
-
Turn One Failed Agent Run Into a Regression Eval Case
Save a failed agent trial as a regression case that keeps the observation, the hypotheses and the expected recovery apart, and learn how many reruns it takes to call a flaky failure fixed.
Notes
-
Pin the Environment Before You Try to Reproduce a Result
Two people can run the same eval script on different library versions and argue about a number neither can reproduce.
-
Reproduce Your Results Table From a Fresh Clone
A results table that only reproduces on your laptop depends on something you never wrote down: an exported variable, a dataset in your home directory, a…
-
Record the Model Snapshot in Every Row of Your Results Table
A saved prompt is not an experiment record. The model snapshot, the sampling settings and the eval set can each move the score, and a results table that…
-
Give Your Ablation Recipe Arguments, Then Check Each Run
You keep typing the same experiment recipe: train with this config and seed, evaluate on the fixed 200-case subset, write the results with the commit hash.
-
Worktrees Separate Files, Not Experiments
You want two agents at once: one refactoring the data loader, one adding an ablation flag. Separate git worktrees give each its own directory and branch, so…
Also in Measure models: Evals and benchmarks, Statistics and experimental design.
Who asks for it
- Research scientists and engineers 68 of 339
- Fellowships, residencies and internships 6 of 32
- Applied AI and forward-deployed 23 of 326
- ML systems and infrastructure 4 of 102
- Machine learning engineers 22 of 143
- Software engineering at labs 39 of 885
Recorded roles that mention it
162 recorded postings mention it. Here are the newest 60 research, fellowship, applied and ML-systems roles; the rest are on the role pages linked above. Each title opens the lab's own posting.
- Software Engineer, Staff: Applied AI, Science & Engineering ↗
Anthropic · Applied AI and forward-deployed · PhD preferred · $405k to $485k base
- Forward Deployed Product Engineer ↗
Cosine · Applied AI and forward-deployed · No degree stated
- Research Scientist ↗
Cursor (Anysphere) · Research scientists and engineers · No degree stated
- AI Research Engineer, Post-Training ↗
Lovable · Research scientists and engineers · No degree stated
- Research Scientist, Asta ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD required · $167k to $261k base
- [Expression of Interest] Research Engineer / Scientist, Alignment - London ↗
Anthropic · Research scientists and engineers · No degree stated
- Machine Learning Infrastructure Engineer, Safeguards Research ↗
Anthropic · ML systems and infrastructure · No degree stated · $350k to $500k base
- Performance Engineer, Inference Engine ↗
Anthropic · ML systems and infrastructure · No degree stated · $350k to $850k base
- Research Engineer / Research Scientist, RL Frontiers ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer / Scientist, Alignment ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $500k base
- Research Engineer, Computer Use ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Pretraining Scaling ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $850k base
- Research Engineer, Pretraining Scaling - London ↗
Anthropic · Research scientists and engineers · No degree stated
- Research Engineer, RL Engineering ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Visual Knowledge Work ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $850k base
- Research Scientist, Interpretability ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $850k base
- Staff+ Research Engineer, RL Data Platform ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Data Infrastructure (Language Modeling) ↗
Cartesia · Research scientists and engineers · No degree stated · $200k to $350k base
- Research Engineer, AI Safety & Alignment ↗
Character.AI · Research scientists and engineers · PhD or equivalent experience · $225k to $400k base
- Member of Technical Staff, Pre-Training Data ↗
Cohere · Research scientists and engineers · No degree stated
- Senior Member of Technical Staff, Synthetic Data ↗
Cohere · Research scientists and engineers · No degree stated
- Research Engineer, Responsibility, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD preferred
- Research Scientist, Gemini Safety and Behavior, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Robotics RL, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Safety Oversight, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Safety Oversight, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist, Verified Code Generation, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD or equivalent experience
- Research Scientist/Engineer, Frontier Reasoning, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD preferred
- Senior Staff Research Scientist/Engineer, Agentic Post Training, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD preferred
- Staff Research Scientist/Engineer, Agentic Post Training, DeepMind ↗
Google DeepMind · Research scientists and engineers · PhD preferred
- ML Research Engineer, London ↗
Isomorphic Labs · Research scientists and engineers · Master's, PhD or equivalent
- Research Leader / Senior Scientist (In vitro / Cellular Pharmacology), Cambridge, MA ↗
Isomorphic Labs · Research scientists and engineers · PhD required
- Research Leader / Senior Scientist (In vivo Pharmacology & Translational Sciences), Cambridge, MA ↗
Isomorphic Labs · Research scientists and engineers · PhD required
- Research Scientist (Applied LLMs), London ↗
Isomorphic Labs · Research scientists and engineers · PhD or equivalent experience
- Research Scientist (Machine Learning), London ↗
Isomorphic Labs · Research scientists and engineers · PhD or equivalent experience
- Liquid Labs - Research Engineer ↗
Liquid AI · Research scientists and engineers · No degree stated
- Member of Technical Staff - GPU Performance Engineer ↗
Liquid AI · ML systems and infrastructure · No degree stated
- Member of Technical Staff - ML Research Engineer, Data ↗
Liquid AI · Research scientists and engineers · PhD required
- Member of Technical Staff - ML Scientist, Japanese Multimodal ↗
Liquid AI · Research scientists and engineers · No degree stated
- Member of Technical Staff, RL Research & Environments ↗
Magic · Research scientists and engineers · No degree stated · $275k to $550k base
- Research Engineer, Language - Wearables Polyglot AI ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · Master's or PhD required
- Research Scientist, Artificial Intelligence ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · PhD or equivalent experience
- AI Engineer, Product ↗
Mistral AI · Applied AI and forward-deployed · No degree stated
- Agent Post-Training Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Artifacts Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Computer Use Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Connectors Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Context Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Agent Post-Training, Frontier Evals and Environments Research ↗
OpenAI · Research scientists and engineers · No degree stated · $380k to $500k base
- Applied AI Engineer ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer - Tokyo ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Codex ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Codex ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Codex ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Codex ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Codex ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Codex | Tokyo ↗
OpenAI · Applied AI and forward-deployed · No degree stated
