Skills · Foundations
Production systems
Taught here
Shipping code that survives: retries, state, monitoring and recovery.
Mentioned in 111 of 339 recorded research roles (33%) and 728 of 1,827 technical roles (40%), in this recorded board.
Everything here that teaches it
10 pieces. Pieces built around this skill come first in each list; the rest use it along the way.
Courses
-
Inference Economics on NVIDIA NIM: Latency, Throughput and Cost per Token
Derive time to first token, decode rate, batching limits and cost per million tokens, then measure them yourself on NVIDIA NIM, hosted or in your own container.
-
Hosted Inference Economics on Nebius Token Factory: Latency and Cost per Task
Derive cost per task from a dated price list, see what rate limits and flavors do to latency, and measure a hosted open-model endpoint yourself with p50 and p95.
Book chapters
-
Chapter 10: Events at random: queues, utilization and scheduled work
Derive how long work waits for one worker under random arrivals (Pollaczek-Khinchine) and measure it with a real model doing the work. Then build durable schedules and stock-event producers.
-
Chapter 13: Slow or dead: leases, fencing and crash recovery
Measure leases on real model turns: false expiry is the latency tail, and detection delay is the lease. Then fence every write by generation and recover from records, not from the model.
-
Chapter 4: Build durable state with SQLite
Store Lucy's stock and the events that explain it in SQLite, so a crash, a replay or a half-applied migration can never leave them disagreeing. Measure when a second writer waits.
-
Chapter 12: Exactly one order: lost replies, retries and idempotency
Derive what retries do when a reply is lost: b/(1 - a - b) duplicate orders, measured over real HTTP. A model re-sends a timed-out order when asked; a stable operation key makes the retry harmless.
-
Chapter 19: What a model call costs, and a deployment that survives
Measure what a model call costs: prefill and decode, memory-bound decoding, the key-value cache from the architecture, batching, latency and cost per task. Then deploy and restore the agent.
Articles
-
A Job ID Is Not a Result: Check the Output Before Marking It Done
A scheduler's job ID, like a deploy hook's 200, says a request was accepted. A five-state model and a small check that marks work done only when the results file exists, from the right checkpoint, with every row.
-
Test an Agent's Fallback Path Against the Same Acceptance Bar
A fallback model runs rarely, so production traffic barely measures it and the aggregate hides it. Derive why, then force failover on your eval set and grade the fallback against the primary's contract.
-
Use a Circuit Breaker When Retries Multiply an Outage
With r retries at each of d layers, one request can become (r + 1)^d sends, and the multiplier peaks exactly when the dependency is down. Derive it, simulate it, and see what a shared breaker changes.
Also in Foundations: Python, Math for ML.
Who asks for it
- Research scientists and engineers 111 of 339
- Fellowships, residencies and internships 5 of 32
- Applied AI and forward-deployed 206 of 326
- ML systems and infrastructure 48 of 102
- Machine learning engineers 48 of 143
- Software engineering at labs 310 of 885
Recorded roles that mention it
728 recorded postings mention it. Here are the newest 60 research, fellowship, applied and ML-systems roles; the rest are on the role pages linked above. Each title opens the lab's own posting.
- Manager, Applied AI Engineering (Megas) ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $320k to $405k base
- Forward Deployed Engineer ↗
Lovable · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineering Lead ↗
Lovable · Applied AI and forward-deployed · No degree stated
- Forward Deployed Creative [UK, French Speaking] ↗
Luma AI · Applied AI and forward-deployed · No degree stated
- Research Engineer, Inference Foundation ↗
Mistral AI · Research scientists and engineers · No degree stated
- Forward Deployed Engineer - London ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - Munich ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - Paris ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - Zurich ↗
OpenAI · Applied AI and forward-deployed · No degree stated
- Member of Technical Staff, ML Platform ↗
Runway · ML systems and infrastructure · No degree stated · $240k to $290k base
- Enterprise Field Engineer – Post-sales ↗
CodeRabbit · Applied AI and forward-deployed · No degree stated · $200k to $250k base
- Senior Applied AI Engineer ↗
CodeRabbit · Applied AI and forward-deployed · No degree stated · $200k to $300k base
- Applied AI Engineer ↗
Cognition · Applied AI and forward-deployed · No degree stated · $180k to $225k base
- Applied AI Engineer - ANZ ↗
Cognition · Applied AI and forward-deployed · No degree stated · $180k to $225k base
- Applied AI Engineer - APAC ↗
Cognition · Applied AI and forward-deployed · No degree stated · $180k to $225k base
- Deployed Engineer ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - Doha ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - EMEA ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - Europe ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - LATAM ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Deployed Engineer - LATAM ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer - APAC ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer - APAC ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Partner Deployed Engineer - Europe ↗
Cognition · Applied AI and forward-deployed · No degree stated
- Research Engineer, Mid-Training ↗
Cognition · Research scientists and engineers · PhD required
- Forward Deployed Engineer ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - EMEA ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Engineer - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Forward Deployed Strategist ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Regional Director, Forward Deployed Engineering ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Regional Director, Forward Deployed Engineering - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- RVP, Forward Deployed Engineering - EMEA ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Software Engineer, Model Routing & Inference ↗
Cursor (Anysphere) · ML systems and infrastructure · No degree stated
- Solutions Architect ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect - India ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect - Japan ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect - Western Europe ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Architect, Central Europe ↗
Cursor (Anysphere) · Applied AI and forward-deployed · No degree stated
- Solutions Engineer, Enterprise, East ↗
Factory · Applied AI and forward-deployed · No degree stated
- Research Engineer ↗
Greptile · Research scientists and engineers · PhD preferred · $180k to $280k base
- Staff Research Engineer (LLM Pre-Training) ↗
JetBrains (Junie, Mellum) · Research scientists and engineers · No degree stated
- AI Research Engineer, Post-Training ↗
Lovable · Research scientists and engineers · No degree stated
- Research Engineer - Provenance (Paris) ↗
Meta (Superintelligence Labs and FAIR) · Research scientists and engineers · No degree stated
- Forward Deployed Engineer ↗
OpenHands (formerly All Hands AI) · Applied AI and forward-deployed · No degree stated · $175k to $240k base
- Machine Learning Scientist ↗
Relace · Research scientists and engineers · Master's, PhD or equivalent
- Forward Deployed Engineer ↗
Replit · Applied AI and forward-deployed · No degree stated · $180k to $300k base
- Senior Applied AI Engineer ↗
StackBlitz (Bolt) · Applied AI and forward-deployed · No degree stated
- Staff Applied AI Engineer ↗
StackBlitz (Bolt) · Applied AI and forward-deployed · No degree stated
- Software Engineer, Inference ↗
Thinking Machines Lab · ML systems and infrastructure · No degree stated · $300k to $400k base
- Forward-Deployed Engineer ↗
Vercel (v0) · Applied AI and forward-deployed · No degree stated
- Senior Research Engineer, Olmo + Molmo ↗
Allen Institute for AI (Ai2) · Research scientists and engineers · PhD or equivalent experience · $174k to $261k base
- Applied AI Architect, Startups ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $240k to $315k base
- Applied AI Architect, Startups ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Architect, Strategic Enterprise Tech ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $240k to $315k base
- Applied AI Engineer ↗
Anthropic · Applied AI and forward-deployed · No degree stated
- Applied AI Engineer, Beneficial Deployments (Life Sciences) ↗
Anthropic · Applied AI and forward-deployed · No degree stated · $280k to $320k base
