Week 0: The Numbers an Inference Engineer Owns
Prerequisite: none. This is a short orientation, not a lesson. After this, you can: name the five numbers this course derives, say which of them it has measured (none on NIM) and decide whether the course is worth your time.
The question behind every serving decision
Put a model behind an API and someone will ask you three things. How long does a user wait? How many requests can one GPU serve? What does a thousand tasks cost? An applied-AI or inference engineer at a lab is expected to answer from first principles and then check the answer with a measurement, not quote a vendor's headline figure. A work sample or interview loop can probe it directly: "Your model's first token takes 900 ms at p95. Where does the time go, and what would you change?"
This course teaches you to answer that question with arithmetic you can show and a measurement you can repeat. NVIDIA NIM is the instrument: an OpenAI-compatible hosted API for many open models, and the same models packaged as containers you can run on your own GPU. That pairing lets you compare a hosted API with self-hosting on one client.
The five numbers
| Number | What it is | Derived from | Lesson |
|---|---|---|---|
| Time to first token (TTFT) | Wait from sending a request to the first streamed token | Prefill: the whole prompt in one pass, plus any queueing | 02 |
| Time per output token (TPOT) | Average gap between tokens once the answer is streaming | Decode: one pass over the weights per token | 02 |
| Throughput | Output tokens per second across all requests | Batch size, key-value cache memory, Little's law | 05 |
| Cost per million tokens | What one million tokens cost you | A price list, or GPU-hours divided by tokens served | 06 |
| Cost per task | What one eval item, agent run or answer costs | Tokens per task times the price, divided by success rate | 06 |
Scroll horizontally to view columns.
Each one is derived in the book first. Chapter 9 derives prefill and decode and measures them on two small local models. Chapter 10 derives how long work waits in a queue. Chapter 19 derives the decode limit, the key-value cache, batching and cost per task. This course applies those derivations to NIM and to a GPU you might rent.
What this course has not done
No NVIDIA API key and no GPU were available while this course was written. So it has not measured a single latency, throughput or cost on NIM, and it does not present one as measured. Where a lesson needs a measured number, it cites the book's runs on small local models and says so. Where it needs a price, it cites the vendor's page and the date the price was read. Everything else is arithmetic on the page or an exercise for you.
That is also the habit the course teaches. NVIDIA's catalog moves quickly: on August 20, 2026
this course's chat lesson named two models that NVIDIA retired six days later, and by September
20 both returned 410 Gone. A latency you measured belongs to one model ID, one endpoint and one
day. Record all three with the number.
The protocol you will run
Lesson 02 gives you a short Python script. It streams requests to any OpenAI-compatible endpoint, records TTFT, TPOT and end-to-end latency per request, and prints p50 and p95. Lesson 05 runs the same script at rising concurrency against your own container. Lesson 06 turns the results into cost per million tokens and a break-even against a hosted price. It follows NVIDIA's own benchmarking definitions, so your numbers can be compared with NVIDIA's.
Why lesson numbering starts at 02
The lessons are numbered 02 through 06. This orientation took the place of a first lesson without renumbering the rest, so older links keep working.
Prefill, decode and time to first token: derive both phases, then time a streaming call to NVIDIA's hosted API and report p50 and p95.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
