Prefill, Decode and Time to First Token on NVIDIA NIM

Why a lab cares about two numbers, not one

"The model takes two seconds" hides two different costs. A request first reads the whole prompt, then writes the answer one token at a time. The two phases hit different hardware limits, so they respond to different fixes. Prompt caching and shorter prompts shorten the first. A smaller or quantized model, or a faster GPU memory, shortens the second. An engineer who reports one blended number cannot say which fix will help, and that is exactly what a latency question in an applied-AI interview tests.

NVIDIA's benchmarking guide (docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html) names the two numbers this lesson uses:

• Time to first token (TTFT): "the time from query submission to the first received token."
• Inter-token latency (ITL), which NVIDIA says is "also known as time per output token (TPOT)": the average gap between tokens, computed as (e2e_latency - TTFT) / (output_tokens - 1).

Derive the two phases

Chapter 9 of the book derives this in full. The short version:

• Prefill runs the model over all P prompt tokens at once. Each token costs about 2N floating-point operations for a model with N parameters, and because every weight read serves all P tokens, prefill is limited by arithmetic.
• Decode writes one token per step. Every step reads every weight, so at batch size one it is limited by how fast memory delivers the weights, not by arithmetic.

Put together, with a queueing delay Q in front:

TTFT ≈ Q + P / prefill_rate
E2E  ≈ TTFT + (O - 1) × TPOT          for O output tokens
TPOT ≥ W / B                          W = weight bytes, B = memory bandwidth (batch size 1)

Bound it for one real model

Take meta/llama-3.1-8b-instruct, the model Lesson 05 runs in a container, on an H100 SXM.

The architecture comes from the configuration file NVIDIA publishes with its FP8 build of the same model (huggingface.co/nvidia/Llama-3.1-8B-Instruct-FP8, config.json): 32 layers, hidden size 4096, 32 attention heads, 8 key-value heads of dimension 128, MLP size 14,336, and a vocabulary of 128,256 with untied input and output embeddings. Counting the matrices:

embeddings in + out   2 × 128,256 × 4,096                         = 1,050.7 M
per layer, attention  4,096×4,096 × 2  +  4,096×1,024 × 2         =    41.9 M
per layer, MLP        3 × 4,096 × 14,336                          =   176.2 M
32 layers × 218.1 M                                               = 6,979.3 M
total (plus a few norm weights)                                   ≈ 8.03 B parameters

NVIDIA's H100 page lists the SXM part at 3.35 TB/s of memory bandwidth and 1,979 teraFLOPS of BF16 tensor throughput "with sparsity," so about 990 teraFLOPS dense. Now the bounds:

BF16 weights  W = 8.03e9 × 2 bytes = 16.06 GB
decode        TPOT ≥ 16.06e9 / 3.35e12 = 4.8 ms   → at most 209 tokens/s per request
FP8 weights   W = 8.03 GB                          → TPOT ≥ 2.4 ms, at most 417 tokens/s
prefill       2 × 8.03e9 × 2,000 tokens / 990e12   ≥ 32 ms for a 2,000-token prompt

These are floors on time, not predictions. Real servers pay fixed costs per step and never reach peak arithmetic. The book measured how far off small models are: on an Apple M4 Pro, decode reached 84 to 128 GB/s of implied weight reads against a published 273 GB/s (Chapter 19). The same runs show the shape the bounds predict: prefill ran 13 to 26 times faster per token than decode. So a 2,000-token prompt and a 256-token answer on this model spend at least 32 ms reading and at least 1.2 s writing. Decode dominates the whole request; prefill and queueing dominate the wait for the first token.

The client: one OpenAI-compatible call

NVIDIA's hosted chat models speak the OpenAI Chat Completions format. Install openai, point it at https://integrate.api.nvidia.com/v1, and put NVIDIA's provider/model-name ID in model:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NVIDIA_API_KEY"],
)

stream = client.chat.completions.create(
    model="nvidia/nemotron-3-super-120b-a12b",
    messages=[{"role": "user", "content": "Explain prefill and decode in two sentences."}],
    temperature=0.5,
    top_p=1,
    max_tokens=1024,
    stream=True,
)
for chunk in stream:
    if not getattr(chunk, "choices", None):
        continue
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

Streaming matters here for a measurement reason, not only a user-experience one. Without it you see only the end-to-end time. With it you can timestamp the first token and the last, and split the request into its two phases.

The same client reaches any hosted chat model by changing model, for example openai/gpt-oss-20b. The swap is one string, and it invalidates every latency you measured. A new model has a different size, a different serving setup and, for a reasoning model, a stream of reasoning tokens before the visible answer. Re-measure after every swap.

When the call fails

These responses were observed from integrate.api.nvidia.com on September 20, 2026, with no API key:

Both retired IDs were the examples in this lesson on August 20, 2026, when a review confirmed them against NVIDIA's reference. A 401 only shows that the request reached a boundary that wants a key. List the IDs the endpoint currently advertises with curl -s https://integrate.api.nvidia.com/v1/models, and confirm a model's capability on its card.

Measure it yourself

NVIDIA's FAQ says hosted access through the Developer Program is "for prototyping, research, development and testing purposes only," and warns that "you may experience extended wait times during periods of high load" (forums.developer.nvidia.com/t/nvidia-nim-faq/300317). That makes the hosted API a fine place to learn the protocol and a poor source of production latency: its TTFT includes a queue you do not control. Record the date and hour of every run.

"""Measure TTFT, time per output token and throughput against an OpenAI-compatible endpoint."""
import argparse, json, math, os, statistics, threading, time
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

def percentile(values, q):
    """Nearest rank: the smallest value that at least q% of observations do not exceed."""
    ordered = sorted(values)
    return ordered[max(1, math.ceil(q / 100 * len(ordered))) - 1]

def one_request(client, args, i):
    prompt = f"[run {i} {time.time_ns()}] " + args.prompt  # a fresh prefix defeats prefix caching
    start = time.perf_counter()
    first, usage, pieces = None, None, 0
    extra = {} if args.no_usage else {"stream_options": {"include_usage": True}}
    stream = client.chat.completions.create(
        model=args.model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=args.max_tokens,
        temperature=0,
        stream=True,
        **extra,
    )
    for chunk in stream:
        if getattr(chunk, "usage", None) is not None:
            usage = chunk.usage
        if not chunk.choices:
            continue
        delta = chunk.choices[0].delta
        text = (delta.content or "") + (getattr(delta, "reasoning_content", None) or "")
        if text:
            pieces += 1
            if first is None:
                first = time.perf_counter()
    end = time.perf_counter()
    out = usage.completion_tokens if usage else None
    ttft = (first or end) - start
    tpot = (end - (first or end)) / (out - 1) if out and out > 1 else None
    return {"i": i, "start": start, "end": end, "ttft_s": ttft, "e2e_s": end - start,
            "tpot_s": tpot, "prompt_tokens": usage.prompt_tokens if usage else None,
            "output_tokens": out, "stream_pieces": pieces}

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--base-url", required=True)
    p.add_argument("--key-env", default="NVIDIA_API_KEY")
    p.add_argument("--model", required=True)
    p.add_argument("--requests", type=int, default=40)
    p.add_argument("--concurrency", type=int, default=1)
    p.add_argument("--max-tokens", type=int, default=256)
    p.add_argument("--prompt", default="Explain the difference between prefill and decode in LLM inference.")
    p.add_argument("--price-in", type=float, help="USD per 1M input tokens, with the date you read it")
    p.add_argument("--price-out", type=float, help="USD per 1M output tokens")
    p.add_argument("--no-usage", action="store_true", help="for endpoints that reject stream_options")
    p.add_argument("--out", default="latency-receipt.jsonl")
    args = p.parse_args()
    client = OpenAI(base_url=args.base_url, api_key=os.environ.get(args.key_env, "not-needed-locally"))

    one_request(client, args, -1)  # warm-up, not recorded
    counter, lock, rows = iter(range(args.requests)), threading.Lock(), []
    def worker():  # closed loop: each worker keeps exactly one request in flight
        while True:
            with lock:
                i = next(counter, None)
            if i is None:
                return
            rows.append(one_request(client, args, i))
    t0 = time.perf_counter()
    with ThreadPoolExecutor(args.concurrency) as pool:
        for future in [pool.submit(worker) for _ in range(args.concurrency)]:
            future.result()  # re-raise a failed request instead of hiding it
    wall = time.perf_counter() - t0

    with open(args.out, "w") as f:
        for r in rows:
            f.write(json.dumps({**r, "model": args.model, "concurrency": args.concurrency}) + "\n")
    ttft = [r["ttft_s"] for r in rows]
    e2e = [r["e2e_s"] for r in rows]
    tpot = [r["tpot_s"] for r in rows if r["tpot_s"]]
    out_tokens = sum(r["output_tokens"] or 0 for r in rows)
    print(f"model {args.model}, {len(rows)} requests at concurrency {args.concurrency}, wall {wall:.1f} s")
    print(f"TTFT  p50 {percentile(ttft, 50):.3f} s   p95 {percentile(ttft, 95):.3f} s")
    print(f"E2E   p50 {percentile(e2e, 50):.3f} s   p95 {percentile(e2e, 95):.3f} s")
    if tpot:
        print(f"TPOT  p50 {1000 * percentile(tpot, 50):.1f} ms  p95 {1000 * percentile(tpot, 95):.1f} ms"
              f"  (decode {1 / percentile(tpot, 50):.0f} tok/s per request at p50)")
    else:
        print("No usage in the stream: TPOT needs token counts; stream pieces are not tokens.")
    print(f"Throughput {out_tokens / wall:.0f} output tok/s, {len(rows) / wall:.2f} req/s")
    print(f"Little's law: {len(rows) / wall:.2f} req/s x {statistics.mean(e2e):.2f} s = "
          f"{len(rows) / wall * statistics.mean(e2e):.2f} in flight (set: {args.concurrency})")
    if args.price_in is not None and args.price_out is not None and all(r["prompt_tokens"] for r in rows):
        cost = [(r["prompt_tokens"] * args.price_in + r["output_tokens"] * args.price_out) / 1e6 for r in rows]
        print(f"Cost per request: mean ${statistics.mean(cost):.6f}, p95 ${percentile(cost, 95):.6f}")

if __name__ == "__main__":
    main()

pip install openai
python measure_latency.py --base-url https://integrate.api.nvidia.com/v1 \
  --model openai/gpt-oss-20b --requests 40 --concurrency 1 --max-tokens 256

View full-size diagram

Exercises

1. Split prefill from decode: Run the script with prompts of about 500, 2,000 and 8,000 tokens at --max-tokens 64. Plot TTFT p50 against prompt length. The slope is roughly the time per prompt token; the intercept is fixed overhead plus queueing.
2. Fit TPOT: Hold the prompt fixed and run --max-tokens at 64, 256 and 1,024. Plot E2E p50 against output tokens. The slope should match the script's TPOT. If it does not, find out why before you trust either.
3. Compare with the bound: For a model whose size and precision you know, compute W / B for a GPU you name. Your measured TPOT on a hosted endpoint will not equal it, and it is not meant to. Write down what the gap could contain: batching with other users' requests, network time, fixed per-step costs, or a different precision than you assumed.

The book's Chapter 19 exercise "Plan memory and latency from the architecture" practices the same arithmetic as a Colab notebook. It is in the course repository at courses/sovereign-agent-book/book/educator/ch19/student/profrod-sovereign-agent-ch19-b-inference-planning-educator-exercise.ipynb.

Revision notes

• September 30, 2026: Repositioned around latency. The lesson used to be "NVIDIA NIM Chat Completion with the OpenAI-Compatible Client," which taught the client and model swaps; its old address now redirects here. Added the prefill and decode derivation, the bounds for Llama 3.1 8B on an H100 SXM, NVIDIA's metric definitions, and the measurement script.
• September 20, 2026: Replaced both example models after NVIDIA retired them on August 26, 2026. Withdrew an August claim that nvidia/llama-3.1-nemotron-70b-instruct was deprecated, which the September 20 checks did not support.