Skills · Foundations
Production systems
Taught here
Shipping code that survives: retries, state, monitoring and recovery.
Mentioned in 112 of 338 open research roles (33%) and 729 of 1,837 technical roles (40%), as of October 2, 2026.
Everything here that teaches it
9 pieces. Pieces built around this skill come first in each list; the rest use it along the way.
Courses
-
Inference Economics on NVIDIA NIM: Latency, Throughput and Cost per Token
Derive time to first token, decode rate, batching limits and cost per million tokens, then measure them yourself on NVIDIA NIM, hosted or in your own container.
-
Hosted Inference Economics on Nebius Token Factory: Latency and Cost per Task
Derive cost per task from a dated price list, see what rate limits and flavors do to latency, and measure a hosted open-model endpoint yourself with p50 and p95.
Book chapters
-
Chapter 10: Events at random: queues, utilization and scheduled work
Derive how long work waits for one worker under random arrivals (Pollaczek-Khinchine) and measure it with a real model doing the work. Then build durable schedules and stock-event producers.
-
Chapter 13: Slow or dead: leases, fencing and crash recovery
Measure leases on real model turns: false expiry is the latency tail, and detection delay is the lease. Then fence every write by generation and recover from records, not from the model.
-
Chapter 4: Build durable state with SQLite
Store Lucy's stock and the events that explain it in SQLite, so a crash, a replay or a half-applied migration can never leave them disagreeing. Measure when a second writer waits.
-
Chapter 12: Exactly one order: lost replies, retries and idempotency
Derive what retries do when a reply is lost: b/(1 - a - b) duplicate orders, measured over real HTTP. A model re-sends a timed-out order when asked; a stable operation key makes the retry harmless.
-
Chapter 19: What a model call costs, and a deployment that survives
Measure what a model call costs: prefill and decode, memory-bound decoding, the key-value cache from the architecture, batching, latency and cost per task. Then deploy and restore the agent.
Articles
-
Test an Agent's Fallback Path Against the Same Acceptance Bar
A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.
-
Use a Circuit Breaker When Retries Multiply an Outage
With r retries at each of d layers, one request can become (r + 1)^d sends, and the multiplier peaks exactly when the dependency is down. Derive it, simulate it, and see what a shared breaker changes.
Also in Foundations: Python, Math for ML.
Who asks for it
- Research scientists and engineers 112 of 338
- Fellowships, residencies and internships 6 of 44
- Applied AI and forward-deployed 207 of 323
- ML systems and infrastructure 50 of 109
- Machine learning engineers 49 of 146
- Software engineering at labs 305 of 877
Open roles that mention it
729 open postings mention it. Here are the newest 60 research, fellowship, applied and ML-systems roles; the rest are on the role pages linked above. Each title opens the lab's own posting.
- Enterprise Field Engineer – Post-sales ↗
CodeRabbit · Applied AI and forward-deployed · No degree stated · $200k to $250k base
- Senior Applied AI Engineer ↗
CodeRabbit · Applied AI and forward-deployed · No degree stated · $200k to $300k base
- Applied AI Engineer ↗
Cognition · Applied AI and forward-deployed · No degree stated · $180k to $225k base
- Applied AI Engineer - ANZ ↗
Cognition · Applied AI and forward-deployed · No degree stated · $180k to $225k base
- Applied AI Engineer - APAC ↗
Cognition · Applied AI and forward-deployed · No degree stated · $180k to $225k base
- Deployed Engineer ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - Doha ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - EMEA ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - Europe ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - LATAM ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - LATAM ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer - APAC ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer - APAC ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer - Europe ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Research Engineer, Mid-Training ↗
Cognition · Research scientists and engineers · PhD required
- Forward Deployed Engineer ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - EMEA ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Regional Director, Forward Deployed Engineering ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Regional Director, Forward Deployed Engineering - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- RVP, Forward Deployed Engineering - EMEA ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Software Engineer, Model Routing & Inference ↗
Cursor (Anysphere) · ML systems and infrastructure · No degree stated
- Solutions Architect ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect - Japan ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect - Western Europe ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect, Central Europe ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Engineer, Enterprise, East ↗
Factory · Applied AI and forward-deployed · No degree stated
- Research Engineer ↗
Greptile · Research scientists and engineers · PhD preferred · $180k to $280k base
- Research Engineer (Agentic Models) ↗
JetBrains (Junie, Mellum) · Research scientists and engineers · No degree stated
- Senior AI Engineer (Core Engine) ↗
JetBrains (Junie, Mellum) · Applied AI and forward-deployed · No degree stated
- Staff Research Engineer (LLM Pre-Training) ↗
JetBrains (Junie, Mellum) · Research scientists and engineers · No degree stated
- Staff/Senior AI Engineer, AI for Code ↗
JetBrains (Junie, Mellum) · Applied AI and forward-deployed · No degree stated
- AI Research Engineer, Post-Training ↗
Lovable · Research scientists and engineers · No degree stated
- Forward Deployed Engineer ↗
Luma AI · Applied AI and forward-deployed · Bachelor's or equivalent · $168k to $252k base
- Research Engineer - Provenance (Paris) ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · No degree stated
- Forward Deployed Engineer - Singapore (Korean Speaking) ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer ↗
OpenHands (formerly All Hands AI) · Applied AI and forward-deployed · No degree stated · $175k to $240k base
- Machine Learning Scientist ↗
Relace · Research scientists and engineers · Master's, PhD or equivalent
- Forward Deployed Engineer ↗
Replit · Applied AI and forward-deployed · No degree stated · $180k to $300k base
- Senior Applied AI Engineer ↗
StackBlitz (Bolt) · Applied AI and forward-deployed · No degree stated
- Staff Applied AI Engineer ↗
StackBlitz (Bolt) · Applied AI and forward-deployed · No degree stated
- Software Engineer, Inference ↗
Thinking Machines Lab · ML systems and infrastructure · No degree stated · $300k to $350k base
- Forward-Deployed Engineer ↗
Vercel (v0) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer ↗
Warp · Applied AI and forward-deployed · No degree stated · $200k to $250k base
- Senior Research Engineer, Olmo + Molmo ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD or equivalent experience · $174k to $261k base
- Applied AI Architect, Startups ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $240k to $315k base
- Applied AI Architect, Startups ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Architect, Strategic Enterprise Tech ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $240k to $315k base
- Applied AI Engineer ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Beneficial Deployments (Life Sciences) ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Beneficial Deployments (Life Sciences) ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $280k to $320k base
- Applied AI Engineer, DNB ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Enterprise ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Enterprise ↗
Anthropic · Applied AI and forward-deployed · No degree stated
