Skip to content

Course in preparation · Coming soon

Distributed Training: Data, Tensor and Pipeline Parallelism

Understand how one model trains across many accelerators. Derive the memory and communication cost of each kind of parallelism, then run each one on several processes on a single machine and compare the numbers.

Who it is for. Engineers moving toward pretraining and infrastructure teams, where postings ask for large-scale distributed training.

The planned syllabus

Planned lessons will help you understand each method, apply it to an AI problem, and judge whether the result holds up.

  1. 01

    Where the memory goes

    Parameters, gradients, optimizer state and activations, counted for a model of a given size.

  2. 02

    Data parallelism and all-reduce

    Collectives derived from first principles, then timed across processes.

  3. 03

    Sharded optimizers

    ZeRO and FSDP: trade communication for memory, and measure the trade.

  4. 04

    Tensor and pipeline parallelism

    Split a layer, split the stack, and work out the pipeline bubble.

  5. 05

    Failure at scale

    Stragglers, checkpoints and restarts: what keeps a long run alive.

Start now with what it builds on

All courses