Course in preparation · Coming soon
Transformer and Inference Internals
Build attention and a transformer block from scratch, then follow a request through inference: sampling, the KV cache, batching, quantization and speculative decoding. Every cost is derived first and measured second.
Who it is for. Candidates for applied-AI, inference and research-engineering roles who need to reason about what a model computes and what serving it costs.
The planned syllabus
Planned lessons will help you understand each method, apply it to an AI problem, and judge whether the result holds up.
- 01
Attention from scratch
Derive scaled dot-product attention and the reason for the square-root scaling; check your implementation against a library's.
- 02
The transformer block
The residual stream, normalization and the MLP; derive parameter and FLOP counts per token and check them against a real model.
- 03
Position
Derive rotary position embeddings as rotations and measure what they do to attention across distance.
- 04
Sampling
Temperature, top-k and top-p, derived as operations on a distribution; measure their effect on entropy and output quality.
- 05
The KV cache
Derive memory per token from layers, heads and head size; measure prefill against decode.
- 06
Batching and throughput
Memory-bound and compute-bound decoding, arithmetic intensity, and tokens per second measured as the batch grows.
- 07
Quantization and speculative decoding
Measure the error low-precision weights introduce, and derive the speed-up speculative decoding can give from its acceptance rate.
