Latency: why a response takes time
The wait before the first token and the pace of later tokens come from different parts of the serving pipeline.
In this lesson
Fast streaming, high throughput, and fast completion are related but different measurements.
Break the wait into stages
A request may wait in a queue, undergo prompt assembly or retrieval, enter model computation, and then stream output. Time to first token measures the delay before the first generated token appears. Time per output token describes the subsequent pace. End-to-end latency includes the whole user-visible task, including external operations.
These definitions make a common experience less mysterious: a system can pause for several seconds, then stream quickly. Another can begin immediately but finish slowly. Neither a single tokens-per-second number nor the first visible word describes the complete experience.
Prefill and decoding
Processing the supplied input is often called prefill. Many positions can be computed together under a causal mask. After that, ordinary autoregressive decoding selects a token, appends it, and computes the next distribution. The newly generated token becomes part of the next step, which creates a sequential dependency.
A key-value cache saves attention intermediates from earlier positions so they need not all be recomputed at each step. Managing this cache efficiently affects serving. The 2023 PagedAttention paper investigates memory organization that supports better use of batched requests. Its reported speedups apply to the evaluated systems, not every possible deployment.
Throughput can compete with responsiveness
Batching lets hardware work on several requests together. This can increase total useful output per second, while scheduling choices influence how long an individual request waits. A server optimized for a large stream of offline jobs can make different tradeoffs from an interactive assistant.
Longer outputs usually require more decoding work. Additional retrieval, tool calls, or model passes can add latency too. Sometimes that extra work improves task success enough to be worthwhile. The right comparison is not simply fastest token production, but acceptable time and cost for a correct completed task.
Measure a realistic workload
Use representative prompt lengths, output lengths, and levels of concurrency. Report a distribution, such as median and a slower percentile, rather than only the best run. Include warmup and caching assumptions. A repeated identical prompt may benefit from caching in a way that a varied real workload does not.
An original planning example separates the terms: 0.4 seconds of queueing, 0.6 seconds of retrieval, 1 second to process the prompt, and 3 seconds of generation sum to 5 seconds. Reducing generation time by 20% changes that stage from 3 to 2.4 seconds and reduces the total to 4.4 seconds, not 4 seconds. Optimizing a component helps only in proportion to its share of the full path.
First token is not completion
System A starts in 0.2 seconds and needs 8 seconds to finish. System B starts in 1 second and needs 3 seconds more. A feels immediate; B completes the task sooner. Report both properties when they matter.
The server with the highest throughput gives every user the shortest wait.
Throughput measures aggregate work. Queueing, scheduling, prefill, and decoding determine individual response time.
Take it with you
- Separate first-token delay from generation pace.
- Measure the complete task including tools.
- Benchmark the expected concurrency and sequence lengths.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Quantization: fewer bits per number
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Efficient Memory Management for Large Language Model Serving with PagedAttention ↗Woosuk Kwon et al. · 2023-09-12 · checked 13 Nov 2026