All sections
Latest
Everything published, newest first.
The log
Entries 1–8 of 8
1 entry
1 entry
Why Attention Divides by the Square Root of the Key Dimension
Setting: in a transformer, attention lets each token weigh every other token. For each token it computes a query vector q, compares it with a key vector k for…Quick thoughttransformersattention1 entry
For a Fixed Compute Budget, a Smaller Model Trained on More Tokens Can Win
Setting: training a large language model costs a fixed amount of computing, and you choose how to split it between a bigger model (more parameters) and more…Quick thoughtscaling-lawspretraining1 entry
LoRA: Train Two Thin Matrices Instead of One Giant One
Setting: fine-tuning adapts a pretrained model to a task by updating its weights. For a very large model, updating all of them means storing a full copy per…Quick thoughtfine-tuningtraining1 entry
The DPO Loss Is One Number: How Much More You Raised the Winner Than the Loser
Setting: to make a language model prefer better answers, you collect pairs: a chosen answer and a rejected answer for the same prompt.Quick thoughtpost-trainingpreference-learning1 entry
A Verifier That Checks Each Step Beats One That Checks Only the Final Answer
Setting: a language model can solve a math problem by writing its reasoning step by step, and you can raise its accuracy by sampling many solutions and…Quick thoughtreasoningverification1 entry
Forward KL Covers, Reverse KL Commits: Which Direction You Distill In
Setting: distillation trains a small student model to imitate a large teacher. The loss that compares them is usually KL divergence, a measure of how far one…Quick thoughttrainingdistillation1 entry
Optimize Against a Reward Model Too Hard and the True Reward Falls
Setting: to train a language model on what people prefer, you fit a reward model, a second model that predicts which of two answers a person would pick, and…Quick thoughtalignmentreward-modelsThat is the first entry in the log.
To browse rather than follow: topics, all articles and all notes. To learn in order: courses and the book.
