Compute: the work behind the words
Training and inference both require computation, but their memory, arithmetic, and throughput demands differ.
In this lesson
Parameter count is only one ingredient in a system’s computational cost.
Count work and storage separately
A parameter is a stored number. A floating-point operation is a piece of arithmetic. Memory capacity determines what can fit; memory bandwidth affects how quickly data can move; compute throughput affects how quickly arithmetic can run. A model can be limited by any of these, depending on its shape and workload.
In an original storage calculation, one billion parameters at two bytes each require two billion bytes for weights alone, about 2 GB in decimal units. That excludes activations, caches, temporary buffers, gradients, and optimizer state. “The weights fit” is therefore not the same as “the complete workload fits.”
Training does more than prediction
Training performs forward computation, computes a loss, propagates derivatives, and updates parameters. Intermediate values and optimization state create additional memory demands. Inference ordinarily omits the backward pass and optimizer, but it may keep a growing key-value cache for generation. The two phases use the same learned structure in different computational workflows.
For resource planning, token count matters alongside model size. Training on more tokens repeats more computation. The relationship between compute budget, parameter count, and data is the subject of the 2022 compute-optimal training study. Its empirical results concern a defined training setting, not an exact billing formula for every architecture.
Attention has its own scaling behavior
In full self-attention, each position can compare with many other positions. A direct attention-score matrix has a number of entries that grows quadratically with sequence length. Doubling a toy sequence from 4 to 8 positions grows the full matrix from 16 to 64 entries. Causal masking removes future connections but does not change that broad quadratic order.
FlashAttention, introduced in 2022, reorganizes exact attention to reduce costly memory traffic and avoid materializing the full attention matrix in the usual way. This is an important distinction: a more efficient implementation need not change the mathematical attention result, and reduced memory use does not automatically make every component linear-time.
Match the measurement to the workload
A hardware peak number is not the speed an application necessarily achieves. Small batches, communication between devices, memory transfers, and software overhead can leave compute units underused. The DeepSpeed Inference research examines how different serving configurations address such constraints. Performance claims belong to their measured setup.
When comparing systems, state model, precision, input length, output length, batch size, hardware, and quality target. A throughput result for many simultaneous requests may not predict the waiting time for one user. A simple resource estimate is useful for orientation; a representative benchmark is needed for an operational decision.
Weight storage is a lower bound
A fictional 3-billion-parameter model with 16-bit weights needs about 6 GB for the raw weights. It can require substantially more total memory once its runtime and context are included. Quantizing the weights changes one term, not every term.
Twice as many parameters always means exactly twice the total cost.
Data length, attention pattern, precision, batching, memory movement, and serving strategy also contribute.
Take it with you
- Separate arithmetic, memory capacity, and bandwidth.
- Training and inference retain different state.
- Report workload conditions with performance numbers.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Latency: why a response takes time
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Training Compute-Optimal Large Language Models ↗Jordan Hoffmann et al. · 2022-03-29 · checked 13 Nov 2026
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ↗Tri Dao et al. · 2022-05-27 · checked 13 Nov 2026
- DeepSpeed Inference ↗Reza Yazdani Aminabadi et al. · 2022-06-30 · checked 13 Nov 2026