Skills · Make it fast and big
Inference and serving
Taught here
Latency, throughput, batching, caching and cost per token.
Mentioned in 127 of 339 recorded research roles (37%) and 653 of 1,827 technical roles (36%), in this recorded board.
Learn it here
- Book, Chapter 9: where the wait goes
- Book, Chapter 19: what a model call costs
- Measure latency: TTFT, decode and the p95 tail
- Course: inference economics on NVIDIA NIM
- Transformer and Inference Internals (coming soon)
The course page has its planned syllabus and a sign-up, which tells us how many people want it.
Everything here that teaches it
15 pieces. Pieces built around this skill come first in each list; the rest use it along the way.
Courses
-
Inference Economics on NVIDIA NIM: Latency, Throughput and Cost per Token
Derive time to first token, decode rate, batching limits and cost per million tokens, then measure them yourself on NVIDIA NIM, hosted or in your own container.
-
Hosted Inference Economics on Nebius Token Factory: Latency and Cost per Task
Derive cost per task from a dated price list, see what rate limits and flavors do to latency, and measure a hosted open-model endpoint yourself with p50 and p95.
-
Voice Agents, Not Chatbots: Latency, Barge-In, and Evaluation
An elective for applied-AI candidates: budget the latency of a voice turn, engineer turn-taking and barge-in, and evaluate a voice agent with error bars and a measurement protocol you run yourself.
Book chapters
-
Chapter 9: Where the wait goes: prefill, decode and a phone channel
Measure where the wait for a reply goes: prefill grows as b n + c n^2, decode is memory-bound, and a prefix cache cuts each turn's prefill. Then connect Telegram requests to durable work and sessions.
-
Chapter 19: What a model call costs, and a deployment that survives
Measure what a model call costs: prefill and decode, memory-bound decoding, the key-value cache from the architecture, batching, latency and cost per task. Then deploy and restore the agent.
Articles
-
Measure Agent Latency: Time to First Token, Decode and the p95 Tail
Measured on small open models you can run on Colab: prefill that grows with the prompt, decode at a fixed cost per token, a prefix cache 22 times faster, and a queue that explodes near capacity.
-
Design a Prompt-Caching Experiment That Measures the Right Thing
Derive when prompt caching pays from Anthropic's write premium and read discount, then run a cold, warm and changed-prefix protocol that keeps cost, context and latency apart.
-
Cost per Solved Task: Break-Even Pass Rates, pass@k and Routing
A cheaper model can raise what each solved task costs. Derive cost per solved task as cost per attempt over pass rate, find the break-even pass rate, and route between models.
Notes
-
Use the Batch API When Your Eval Can Wait a Day
A nightly sweep over 10,000 eval items and the debugging session where you rerun five failures have different deadlines.
-
Put the Rubric First, Then Measure Your Prompt-Cache Hits
Grading 2,000 eval items with the same rubric and the same few-shot examples means sending the same long prefix 2,000 times.
-
Give a Reasoning Model the Budget and the Success Criteria
OpenAI's reasoning guide gives short advice. Keep prompts simple and direct. Skip "think step by step," because these models reason internally.
-
Ask Your LLM Grader for the Smallest Output That Still Carries Evidence
An LLM grader in an eval pipeline often writes a paragraph of rationale for every item. Across a few thousand items, that paragraph is most of your wait, and…
-
Time Each Stage of an Agent Run Before You Optimize It
An eval sweep that takes all night limits how many ablations you can run in a week, and "why is this agent slow?" is a fair interview question for an…
-
Compare Cost per Accepted Result, Not per Token or per Session
A cheaper model, or a cheaper Claude Code setup, can cost more per result you keep. A call or a session whose output you throw away cost money and bought…
-
The 7x Agent-Team Figure Applies to Plan Mode: Measure Your Own Ratio
The slug of this note says agent teams cost seven times more. The documented claim is narrower, and the difference matters when you plan compute or explain a…
Also in Make it fast and big: PyTorch, JAX, Distributed training, GPU kernels.
Who asks for it
- Research scientists and engineers 127 of 339
- Fellowships, residencies and internships 8 of 32
- Applied AI and forward-deployed 100 of 326
- ML systems and infrastructure 82 of 102
- Machine learning engineers 82 of 143
- Software engineering at labs 254 of 885
Recorded roles that mention it
653 recorded postings mention it. Here are the newest 60 research, fellowship, applied and ML-systems roles; the rest are on the role pages linked above. Each title opens the lab's own posting.
- Applied AI Architect ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Manager, Applied AI Engineering (Megas) ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $320k to $405k base
- Staff Research Engineer, Multi-Agent Scaling ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer - Eval Platform ↗
Mistral AI · Research scientists and engineers · Master's, PhD or equivalent
- Research Engineer, Inference Foundation ↗
Mistral AI · Research scientists and engineers · No degree stated
- Member of Technical Staff, ML Platform ↗
Runway · ML systems and infrastructure · No degree stated · $240k to $290k base
- Research Engineer, ML Infrastructure ↗
Cognition · Research scientists and engineers · PhD required
- Field Engineer ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Field Engineer - ANZ ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Field Engineer - ASEAN ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Field Engineer - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Field Engineer - South Korea ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Field Engineering - Japan ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - EMEA ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Software Engineer, ML Infrastructure ↗
Cursor (Anysphere) · ML systems and infrastructure · No degree stated
- Software Engineer, ML Platform ↗
Cursor (Anysphere) · ML systems and infrastructure · No degree stated
- Software Engineer, Model Routing & Inference ↗
Cursor (Anysphere) · ML systems and infrastructure · No degree stated
- Staff Research Engineer (LLM Pre-Training) ↗
JetBrains (Junie, Mellum) · Research scientists and engineers · No degree stated
- AI Research Engineer, Post-Training ↗
Lovable · Research scientists and engineers · No degree stated
- Senior Applied AI Engineer ↗
StackBlitz (Bolt) · Applied AI and forward-deployed · No degree stated
- Software Engineer, Inference ↗
Thinking Machines Lab · ML systems and infrastructure · No degree stated · $300k to $400k base
- Solutions Architect ↗
Vercel (v0) · Applied AI and forward-deployed · No degree stated · $190k to $340k base
- Senior Research Engineer, Olmo + Molmo ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD or equivalent experience · $174k to $261k base
- Applied AI Architect ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Architect ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Architect, Cyber ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $240k to $315k base
- Engineering Manager, GPU (ML Accelerator) ↗
Anthropic · ML systems and infrastructure · No degree stated · $500k to $850k base
- Engineering Manager, Inference Infrastructure ↗
Anthropic · ML systems and infrastructure · No degree stated · $405k to $625k base
- Machine Learning Infrastructure Engineer, Safeguards Research ↗
Anthropic · ML systems and infrastructure · No degree stated · $350k to $500k base
- Performance Engineer ↗
Anthropic · ML systems and infrastructure · No degree stated · $280k to $850k base
- Performance Engineer, GPU ↗
Anthropic · ML systems and infrastructure · No degree stated · $280k to $850k base
- Performance Engineer, Inference Engine ↗
Anthropic · ML systems and infrastructure · No degree stated · $350k to $850k base
- Performance Engineer, Inference Systems ↗
Anthropic · ML systems and infrastructure · No degree stated · $350k to $850k base
- Research Engineer / Performance Engineer, RL Distributed Systems ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer / Research Scientist, RL Frontiers ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer / Research Scientist, Tokens ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $500k base
- Research Engineer, Chip Design RL (Reinforcement Learning) ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Discovery ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $850k base
- Research Engineer, Interpretability ↗
Anthropic · Research scientists and engineers · No degree stated · $315k to $560k base
- Research Engineer, Machine Learning (RL Velocity) ↗
Anthropic · Research scientists and engineers · No degree stated
- Research Engineer, Machine Learning (RL Velocity) ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer, Pretraining ↗
Anthropic · Research scientists and engineers · Master's or PhD required
- Research Engineer, RL Engineering ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
- Research Engineer/Research Scientist, Pre-training ↗
Anthropic · Research scientists and engineers · Master's or PhD required · $350k to $850k base
- Staff + Senior Software Engineer, Inference ↗
Anthropic · ML systems and infrastructure · No degree stated
- Staff + Senior Software Engineer, Inference Deployment ↗
Anthropic · ML systems and infrastructure · No degree stated · $320k to $485k base
- Staff + Senior Software Engineer, Inference Infrastructure ↗
Anthropic · ML systems and infrastructure · No degree stated · $320k to $485k base
- Staff + Sr. Software Engineer, Cloud Inference ↗
Anthropic · ML systems and infrastructure · No degree stated · $320k to $485k base
- Staff + Sr. Software Engineer, Cloud Inference Launch Engineering ↗
Anthropic · ML systems and infrastructure · No degree stated · $320k to $485k base
- Staff Infrastructure Engineer, Cluster Infrastructure ↗
Anthropic · ML systems and infrastructure · No degree stated
- Staff Research Engineer, Discovery Team ↗
Anthropic · Research scientists and engineers · No degree stated · $350k to $850k base
- Staff Software Engineer, Inference ↗
Anthropic · ML systems and infrastructure · No degree stated
- Staff+ Infrastructure Engineer, Cluster Infrastructure ↗
Anthropic · ML systems and infrastructure · No degree stated · $320k to $485k base
- Staff+ Research Engineer, RL Data Platform ↗
Anthropic · Research scientists and engineers · No degree stated · $500k to $850k base
