Course in preparation · Coming soon
Transformer and Inference Internals
Build attention and a transformer block from scratch, then follow a request through inference: sampling, the KV cache, batching, quantization and speculative decoding. Every cost is derived first and measured second.
Who it is for. Candidates for applied-AI, inference and research-engineering roles who need to reason about what a model computes and what serving it costs.
The planned syllabus
Every module will meet the book's bar before it opens: the math derived on the page, a Colab notebook on Python 3.12, and measurements from real models.
- 01
Attention from scratch
Derive scaled dot-product attention and the reason for the square-root scaling; check your implementation against a library's.
- 02
The transformer block
The residual stream, normalization and the MLP; derive parameter and FLOP counts per token and check them against a real model.
- 03
Position
Derive rotary position embeddings as rotations and measure what they do to attention across distance.
- 04
Sampling
Temperature, top-k and top-p, derived as operations on a distribution; measure their effect on entropy and output quality.
- 05
The KV cache
Derive memory per token from layers, heads and head size; measure prefill against decode.
- 06
Batching and throughput
Memory-bound and compute-bound decoding, arithmetic intensity, and tokens per second measured as the batch grows.
- 07
Quantization and speculative decoding
Measure the error low-precision weights introduce, and derive the speed-up speculative decoding can give from its acceptance rate.
