Measure Agent Latency: Time to First Token, Decode and the p95 Tail

On a laptop GPU, Qwen2.5 1.5B takes 1.29 seconds to read a 2,048-token prompt and 22 milliseconds for each token it writes after that. Keep the first 2,048 tokens cached, and the first token of the answer arrives in 62 milliseconds. Put one worker behind a queue at 95% utilization, and the average request spends 5.5 seconds in the system for 0.9 seconds of work.

Those four numbers are the four places a user's wait goes: prefill, decode, a cache, and a queue. This article derives each one and measures it, and shows how many trials a tail percentile needs before you can quote it. The same accounting applies to any model you serve or call.

Everything below ran on 2026-09-30, with one rerun on 2026-10-01, on two small open models, Qwen2.5-Instruct at 0.5B and 1.5B parameters, in float16 on an Apple M4 Pro GPU, with no API and no cost. The same code runs on a free Colab GPU, where every number will be different and every shape should be the same. The code, every timing, the statistics and a notebook are in profrodai-resources. Chapter 9 of the book derives prefill and decode for the agent it builds.

Top: measured time to first token (median dots, p95 ticks) with the fitted curve; dashed is its straight-line part. Bottom: one request you size, split into prefill and decode.

Qwen2.5 0.5BQwen2.5 1.5B

View full-size figure

Qwen2.5 0.5B, reading the fitted curve: a 2,048-token prompt reaches its first token in 489 ms, and 200 tokens of reply add 3,193 ms at about 62 tokens per second. Decode is 87% of this wait. The quadratic term is 8% of prefill at this length.

prompt tokens: 2,048reply tokens: 200

Prefill and decode

A reply is produced in two phases. Prefill reads the whole prompt in one forward pass. All its tokens are known in advance, so the model processes them together, and the cost grows with the prompt. Decode writes the reply one token at a time, because each token depends on the one before. For a prompt of $P$ tokens and a reply of $m$ tokens,

where $T_0$ collects everything outside the model: network, queue and tokenization. Here the model runs locally in a hand-written loop, so $T_0$ is near zero and each phase is timed on its own, with the GPU synchronized before every clock read.

Prefill should cost roughly $a + bP + cP^2$. The linear term is the work done per token (the matrix multiplications), and the quadratic term is attention, where every token looks at every earlier one. Fitted over 140 trials from 64 to 4,096 tokens, the 1.5B model gives $b = 0.59$ ms per token, and the quadratic term is 10% of prefill at 4,096 tokens. So doubling the prompt from 2,048 to 4,096 tokens takes the median from 1.29 to 2.71 seconds, 2.09 times as long. In the figure, the gap between the solid curve and the dashed straight line is the quadratic term.

Decode behaves differently. The 1.5B model took a median of 22 ms per step after a 128-token prompt and 22.5 ms after a 2,048-token prompt, so context length barely mattered at these sizes. Model size mattered less than you might expect. The 0.5B model has a third of the weights, and it prefilled 2,048 tokens 2.6 times faster (489 ms), but it decoded only 1.1 to 1.4 times faster, at 16 to 19 ms per step. A single-sequence decode step on a small model spends much of its time on fixed costs per step rather than on arithmetic. Batching several requests is how servers recover that time; this run does not batch.

The split decides what to optimize. Here are two request shapes on the 1.5B model:

For the chat answer, streaming lets the reader start at 1.3 seconds, and decode speed is what they feel. An agent step reads a long transcript and writes a short decision, so for agents the prompt is the cost.

A prefix cache, measured

Prefill of a shared prefix is repeated work. If every request starts with the same 2,048 tokens of instructions and tools, the keys and values the model computes for those positions are the same every time. Store them once, and a request with that exact prefix only has to prefill its own question.

The experiment runs three conditions, interleaved, 200 trials each. Each trial is a 2,048-token prefix followed by a 32-token question.

• Cold: no cache. The model prefills all 2,080 tokens.
• Warm: the prefix's key-value cache comes from a store keyed on its exact tokens. The model prefills only the 32-token question.
• Changed: the prefix differs in its first token. The lookup misses and the model prefills everything.

Time to first token for a 2,048-token shared prefix and a 32-token question, every trial a dot (log scale). Tall tick: p50. Short tick and band: p95 and its 95% interval.

Qwen2.5 0.5BQwen2.5 1.5B

View full-size figure

Qwen2.5 0.5B, 200 trials each: reusing the prefix's cache cuts the median from 482 ms to 27.3 ms, 18 times faster, and the p95 from 534 ms to 101 ms. Changing only the first token of the prefix misses the cache: 485 ms, the same as cold. On 20 questions the warm and cold next token agreed 20 times.

On the 1.5B model the median time to first token falls from 1,368 ms cold to 62 ms warm, 22 times faster. The p95 falls from 1,603 to 72 ms. One changed token at the start of the prefix gives 1,352 ms, the same as cold, because the cache stores computation for exact positions: change token one, and every key and value after it is different. That is why production prompt caches match on exact prefixes, and why anything that varies (a timestamp, a user name, a reordered tool list) belongs after the cached part.

Two checks keep the speedup honest. On 20 questions, the warm path's next token matched the cold path's every time, and the logits differed by at most 0.03, which is float16 rounding. And the warm tail is not the cache. On the 0.5B model the warm median was 27 ms but the p95 was 101 ms, with a 95% interval of 71 to 166 ms. When the work is that short, a few slow trials caused by the machine rather than the model set the tail. For what caching costs and saves on a hosted API, where the provider bills cache writes and reads, see the caching experiment.

How many trials a percentile needs

Averages hide slow requests, so latency is reported by percentiles. Take the qth percentile as the smallest observed value that at least q% of the observations do not exceed. The median, p50, describes a typical request. The p95 describes the tail that a service promise usually targets.

A percentile from a sample is an estimate, and you can put an interval on it without assuming any distribution. The number of observations below the true 95th percentile is binomial with $n$ trials and probability 0.95. So the sorted sample's $l$th and $u$th values bracket the true p95 whenever

Solving for the tightest ranks gives:

With 30 trials you cannot bound a p95 at all; the number you would quote is the slowest or second-slowest run. That is why each cache condition above has 200 trials, and why its p95 comes with an interval. Plan for a few hundred trials per condition before you quote a tail, and report $n$ with every percentile.

Queueing: Little's law, and why waits explode

Once requests share a limited resource, some of them wait. Little's law relates three long-run averages: $L$, the number of requests in the system; $\lambda$, the arrival rate; and $W$, the mean time each request spends in the system.

The derivation fits in a line. Over a long window of length $T$, add up the time every request spent in the system. That area equals the time-average number in the system times $T$. It also equals the number of arrivals, about $\lambda T$, times the average stay $W$. Divide by $T$.

Little's law says nothing about how large $W$ gets. For one server with random (Poisson) arrivals and service time $S$, the Pollaczek–Khinchine formula does:

The second term is the wait in line, and its denominator $(1 - \rho)$ is why queues explode near capacity. When service times barely vary, as here ($E[S^2]$ within 3% of $E[S]^2$), the line adds $\rho / (2(1-\rho))$ service times: half of one at $\rho = 0.5$, four and a half at $\rho = 0.9$, and without bound at $\rho = 1$.

To test both, the run puts the model behind a single worker. Each request is a 256-token prompt and a 32-token reply, about 0.9 seconds of work on the 1.5B model. Requests arrive at random at five rates, 200 at each. A separate monitor counts the requests in the system every 5 ms, so $L$ is measured independently of $\lambda$ and $W$.

One worker serving the model, random arrivals. Dots: measured mean time in system. Curve: the queueing formula, from the measured service times.

Qwen2.5 0.5BQwen2.5 1.5B

ρ ≈ 0.30ρ ≈ 0.52ρ ≈ 0.60ρ ≈ 0.71ρ ≈ 0.82

View full-size figure

Qwen2.5 0.5B at ρ ≈ 0.60 (200 requests, service 456 ms): mean time in system 0.75 s, against 0.79 s from the formula. Little's law: sampling the queue every 5 ms gave L = 0.98 requests in the system; λ·W = 0.99.

On the 1.5B model, the measured mean time in system follows the formula up to $\rho \approx 0.8$: 1.26, 1.43, 2.04 and 2.42 seconds, against predictions of 1.23, 1.47, 1.95 and 2.58. At $\rho = 0.95$ the measured mean was 5.5 seconds, six times the service time, while the formula predicts 10.2. Near capacity, 200 requests are not enough for the queue to reach its long-run average: the run ended before the line had grown to its steady-state length. One request in 20 spent more than 10 seconds in the system. On the 0.5B model, with requests of about 0.47 seconds, the measured means stayed within 11% of the formula at every level. That GPU served faster under steady load (449 ms per request at the top level against 493 ms at the lowest), so its top level reached ρ = 0.82 rather than 0.9. Little's law held at every level: the monitor's $L$ and $\lambda W$ agreed within 1.6%, from 0.46 against 0.46 to 5.82 against 5.86.

Use Little's law in both directions when you size an agent service. If requests arrive at 2 per second and each spends 4 seconds in the system, 8 are in flight on average, so a concurrency limit of 5 caps throughput at 5 / 4 = 1.25 per second and the queue grows. The formula adds the second warning: a limit you meet on average is already too small, because at $\rho$ near 1 the wait is dominated by the line.

The agent's timeline

An agent turn has more boundaries than a model call. The request arrives and may wait in a queue. Then context is assembled, the model starts, and a first token is emitted, which may be a tool call rather than visible text. Then it waits for tools, and finally produces a result someone accepts.

View full-size diagram

flowchart LR
    A["Request arrives"] --> B["Queue exit"]
    B --> C["Context assembled"]
    C --> M["Model starts"]
    M --> D["First visible token"]
    D --> E["Tool call ends"]
    E --> F["Result accepted"]

Each arrow is a segment you can timestamp, and each measurement above is one of them. They interact. A prefix cache that cuts time to first token by a factor of 22 also sends tool calls to the tool stage sooner. If that stage is a shared pool near capacity, the queueing result says its wait can grow faster than the model's time shrank. Report the model's latency and the time to an accepted result separately, because they can move in opposite directions.

What would change these conclusions

• The machine: a laptop GPU in float16, running one sequence at a time. A data-center GPU with batching changes every absolute number. Batching in particular changes decode, where many sequences share each read of the weights.
• The engine: a hand-written loop over Hugging Face transformers. Serving engines with paged attention and fused kernels shrink the fixed per-step cost that dominated decode here.
• Model size: at 0.5B and 1.5B parameters, attention is a modest share of prefill even at 4,096 tokens. Longer contexts and larger models make the quadratic term matter more.
• The queue: one worker, first come first served, 200 requests per level. Real services have several workers and priorities, and a run near capacity needs far more requests than 200 to reach its long-run average.

Why this matters in a lab

Inference and serving questions are standard in applied-AI interviews, and so is the trap: an unstated choice of which interval counts as "the latency." The strong answers have the parts above. Prefill grows with the prompt and has a quadratic term. Decode is one step per token and, at batch size one, is dominated by per-step cost. A prefix cache only helps on exact prefixes. A tail percentile needs a few hundred trials. And Little's law plus the $\rho/(1-\rho)$ term tells you how much headroom a service needs.

A question that separates candidates: your agent serves 50 requests per second, and each takes 2 seconds of model time on one replica. How many replicas do you need?