Course in preparation · Coming soon
Pretraining and Scaling Laws
Predict a big run from small ones. Train a family of tiny models, fit a scaling law to their losses, and measure how far the fit can be trusted before you spend the compute.
Who it is for. Researchers and engineers aiming at pretraining teams, where postings ask about data mixtures, compute budgets and scaling.
The planned syllabus
Planned lessons will help you understand each method, apply it to an AI problem, and judge whether the result holds up.
- 01
Tokens and data
Tokenizers, deduplication and filtering: what goes into a pretraining corpus.
- 02
Compute accounting
FLOPs per token, model size and data size, derived and checked against a run.
- 03
Fitting a scaling law
A sweep of tiny models, a power-law fit and its confidence interval.
- 04
Data mixtures
Change the mixture, measure the loss on each domain, and see what transfers.
- 05
When the curve breaks
Instabilities, learning-rate sensitivity and what small runs cannot tell you.
