Inference Economics on NVIDIA NIM: Latency, Throughput and Cost per Token
Where do the time and the money go when a model answers? This course derives each number from prefill and decode, the key-value cache, Little's law and dated prices, using NVIDIA NIM: hosted models, embedding and reranking, image and video containers priced per denoising step, and a self-hosted container under load. Each lesson gives a p50 and p95 protocol. The course has not run it (no NVIDIA API key or GPU was available), so no NIM number is measured; measured numbers come from the book's runs on small local models. API samples were checked against NVIDIA's documentation on September 20, 2026, and pricing and benchmarking sources on September 30.
ToolChecked against NVIDIA NIM NIM LLM 2.0.12 on 2026-09-20.
Rod Rivera
Professor
Desk
Every lesson is free. Lessons marked open need no account; the rest open once you register.
What You'll Learn
- Derive time to first token and time per output token from prefill and decode, and bound a model's decode rate from its weight size and the GPU's memory bandwidth
- Time a streaming call to NVIDIA's hosted NIM API and report time to first token, time per output token and end-to-end latency as p50 and p95 over enough requests to mean something
- Put retrieval, vision-language and diffusion calls into the same latency and cost budget, and name each one's cost unit: tokens, passages, image tokens or denoising steps
- Size the key-value cache from a model's configuration, and explain why cache memory, not arithmetic, usually caps how many requests a GPU can batch
- Sweep concurrency against a self-hosted NIM container, read the throughput-latency curve, and check the result with Little's law
- Compute cost per million tokens and cost per task for a hosted API and a self-hosted GPU, and find the throughput and utilization at which self-hosting breaks even
Prerequisites
- Comfort reading Python and basic algebra; every formula is derived on the page
- For the hosted measurements: an NVIDIA API key from build.nvidia.com. NVIDIA's FAQ describes this access as free for prototyping and development, not for production
- For Lessons 04 and 05's container steps: an NVIDIA GPU, Docker with the NVIDIA Container Toolkit, and an NGC API key. Without them you can still do the arithmetic and read the protocol
- Helpful, not required: the book's chapters on latency (/book/ch09-messaging), queues (/book/ch10-scheduling) and cost (/book/ch19-operation)
Syllabus
Read the rest of Inference Economics on NVIDIA NIM: Latency, Throughput and Cost per Token
The first lessons need no account. Sign in free to read the rest and keep your place on any device.
