The math path
Work through vectors, dot products, normalized exponentials, and gradients. The labs keep the calculations small enough to inspect.
By the endCompute a toy attention row and explain how its normalization differs from a loss gradient.
Start path: Embeddings: giving tokens coordinates →7 lessons · 42 min total
Mark lessons as read to save your place on this browser.
- 16 MIN READING · Not read yet
Embeddings: giving tokens coordinates
Compare vector direction with cosine similarity.
Read the lesson → - 26 MIN READING · Not read yet
Neural networks: composing simple calculations
Connect weighted sums to nonlinear transformations.
Read the lesson → - 36 MIN READING · Not read yet
Attention: a computed weighted mixture
Calculate query–key compatibility scores.
Read the lesson → - 46 MIN READING · Not read yet
Causal masking: no peeking at the answer
Remove disallowed positions before normalization.
Read the lesson → - 56 MIN READING · Not read yet
Softmax: turning scores into a distribution
Convert scores into a distribution.
Read the lesson → - 66 MIN READING · Not read yet
Loss: a number to learn from
Score the target rather than the selected sample.
Read the lesson → - 76 MIN READING · Not read yet
Gradient descent: changing the weights
Move parameters using derivatives.
Read the lesson →