PART 01 // OVERVIEW
6 MIN READ
EPOCHS 1 → 45+
Start with two numbers (a slope w and an offset b) trying to fit noisy points. Follow the four architectural walls that take a 1958 perceptron to a modern Sparse Mixture-of-Experts Transformer.
PART 02 // NON-LINEARITY
5 MIN READ
BLOCKER: CEILING // σ REQ
Stacking fifty linear layers still gives you one straight line. Why Minsky & Papert's 1969 XOR proof stalled neural networks for a decade, why Sigmoid chokes deep gradients, and why max(0, x) fixed it.
PART 03 // GENERALIZATION
6 MIN READ
BLOCKER: OVERFIT // VAL GAP
Why adding more parameters to a small dataset makes your test score worse. How the 2σ²/N estimation tax works, why data augmentation cures it, and what DeepMind's 2022 Chinchilla paper proved about data-starved LLMs.
PART 04 // OPTIMIZATION
6 MIN READ
PRESTIGE TIER 1 // SGD → ADAM
Plain gradient descent zig-zags across narrow valleys. How Polyak's 1964 heavy-ball momentum and Hinton's 2012 Coursera lecture slide combined into Adam - the loop that trained GPT-3.
PART 05 // ARCHITECTURES
7 MIN READ
BLOCKER: SEGMENTED & ROUTING HEADS
Convolutional kernels scan small local windows and share weights across space. That works for edges in photos, and breaks when word #4 depends on word #400. How softmax(QKᵀ/√d)V routes by content instead of distance.