PART 04 // THE OPTIMIZER TREE

SGD, Momentum, and Adam: What Optimizers Actually Do on a Loss Surface

Stepping straight downhill sounds optimal until the hill is a narrow canyon. Here is why it took sixty years to go from Robbins & Monro (1951) to Kingma & Ba (2014).

OPTIMIZERS (∇) 6 MIN READ BY LOYAL LOGIC

If you switch the visualizer on the Train screen of Epoch Idle from SCATTER to LANDSCAPE, the data points disappear. Instead, you look straight down at a topographic contour map of your error L(w, b), with weight w on the horizontal axis, bias b on the vertical axis, and a glowing ball rolling toward the dark valley floor.

That bowl is the loss surface. And how your model gets from the rim of that bowl to the minimum at the bottom is the job of the optimizer.

1. Vanilla SGD: The Ravine Problem (1951)

Stochastic Gradient Descent (SGD) computes the slope of the loss at your current spot and steps directly against it:

θ ← θ − η · g g = ∇L(θ) is the current gradient vector, and η is the learning rate.

If the loss bowl is a perfect circle, pointing straight down the slope points straight at the center. But real neural network loss surfaces are almost never circular. One parameter might change the output delicately while another parameter swings it violently. Geometrically, that stretches the bowl into a steep, narrow canyon (an ill-conditioned Hessian).

When you stand on the wall of a narrow canyon, which way is steepest downhill? Across the canyon toward the opposite wall, not along the canyon floor toward the exit. So plain SGD ricochets back and forth from wall to wall while making agonizingly slow progress down the valley. If you crank up the learning rate η to hurry it along, the steps overshoot the opposite wall and bounce right out of the canyon.

Epoch Idle Optimize screen showing the SGD to Momentum, RMSProp, and Adam optimizer tree
THE OPTIMIZER TREE // SGD BRANCHES INTO MOMENTUM AND RMSPROP BEFORE COMPILING INTO ADAM

2. Momentum: The Heavy Ball (1964)

In 1964, Boris Polyak proposed a simple physical fix: give the point mass, like a heavy iron ball rolling down the canyon.

v ← γ · v + η · g
θ ← θ − v v is velocity. Standard decay is γ = 0.90, keeping 90% of the previous step's direction.

Think about what happens to that rolling ball in a narrow canyon. The cross-canyon gradients point positive on one step, negative on the next, positive on the next - so when you average them into velocity v, the side-to-side bounces cancel each other out. Meanwhile, the small component pointing steadily down the canyon floor points the same way every single step, so it compounds and accelerates.

3. RMSProp: Per-Parameter Step Sizes (2012)

Momentum fixes direction; RMSProp fixes scale. One of machine learning's oddest trivia facts is that RMSProp was never published in a peer-reviewed paper. Geoffrey Hinton introduced it on Slide 29 of Lecture 6e in his 2012 Coursera class, and researchers started citing the slide URL directly because it worked so well.

RMSProp keeps a running average of the squared gradient for every individual parameter, and divides the step by its square root:

s ← β · s + (1 − β) · g²
θ ← θ − η · g / (√s + ε) Parameters with huge, noisy gradients get divided by a large √s (shrinking their step). Quiet parameters get divided by a tiny √s (stretching their step).

By rescaling each axis according to its own recent steepness, RMSProp stretches the narrow canyon back toward a round bowl.

4. Adam (2014) and Lion (2023)

Once you understand Momentum and RMSProp, you already understand Adam (Adaptive Moment Estimation, Kingma & Ba, 2014). Adam simply runs both ideas at once:

Adam became the universal default trainer for transformers - GPT-3 trained with Adam at β₁ = 0.9, β₂ = 0.95. Its only downside at frontier scale is memory: storing both m and v for a 175-billion-parameter model doubles your optimizer state RAM. That pressure led Google Brain in 2023 to use evolutionary program search to discover Lion, which discards the second moment v entirely and steps every parameter by the sign() of its momentum.

IN EPOCH IDLE // PRESTIGE TIER 1

In Epoch Idle, executing a Backprop Step converts your run into ∇ Gradients to spend down the historical tree: SGD splits into Momentum (which automates weight & bias steps) and RMSProp (which automates adaptive learning rate η). Maxing both branches lets you compile Adam at the apex, automating the entire training stack and opening Lion beyond it.

AUTOMATE DESCENT // FREE ON ANDROID & iOS

Climb the SGD → Adam → Lion tree

Watch the −∇L descent trail on a live contour map and unlock full optimizer automation in Epoch Idle.