PART 03 // GENERALIZATION & DATA

Overfitting Without the Calculus: Why Bigger Models Fail on Small Data

Give a model enough free dials and it stops learning the underlying rule. It just writes down the noise in the room.

OVERFITTING (VAL GAP) 6 MIN READ BY LOYAL LOGIC

In almost every incremental game, buying more upgrades makes your numbers go up. When players reach Epoch 12 in Epoch Idle, they unlock Convolutional Kernels, excitedly buy six of them in a row, and watch their progress grind to a halt behind an amber warning: OVERFIT // VAL GAP.

That stall isn't an arbitrary cooldown. It is the exact statistical penalty every machine learning engineer fights when a model grows faster than its dataset.

Two Points Define Any Line

Imagine a hidden true trend y = 2x, plus a little random measurement jitter of size σ on each observation. Suppose you collect 16 noisy points across the graph.

If you fit a single straight line (2 parameters: one slope w and one offset b) across all 16 points, the random positive and negative jitters mostly cancel each other out. One point sits a little high, the next sits a little low, and the line cuts cleanly down the middle.

Now split those same 16 points among 8 separate local kernels. Each kernel only sees 2 points in its window. And how many points does it take for a 2-parameter line to hit every point in its window with zero error? Two.

Each kernel connects its two noisy dots directly. On the points you trained on (training loss), the error looks fantastic. But the moment a new data point arrives from the same distribution (validation loss), your zig-zagging chain of lines misses badly because it chased the random jitter of those specific 16 samples.

The Exact Price of a Parameter: 2σ² / N

In ordinary least-squares regression, you don't need a simulation to guess how much worse a model gets on held-out data when you add parameters. There is an exact law:

Expected Held-Out Excess Error ≈ p · σ² / N Each local expert estimates p = 2 parameters (slope and intercept) from N effective samples, adding 2σ² / N of estimation variance on unseen data.

Look at that fraction: (experts × 2σ²) / N. Every time you add a kernel or an attention head, the numerator grows. If the denominator (N, the amount of training data) stays small, your held-out validation loss climbs above the convergence threshold even when your parameters are fully trained.

IN EPOCH IDLE // TRAIN VS. VAL LOSS

When you add kernels or heads in Epoch Idle, the simulation tracks both your fit to the underlying truth and the 2σ² / N estimation penalty per expert. If your stack is oversized for your current sample budget, the formula bar flags OVERFIT // VAL GAP and publishes the paper Pitfall: Overfitting.

How Labs Fix It: Data Augmentation & Chinchilla

When your numerator (parameters) is too big for your denominator (data), you only have two honest moves:

  1. Shrink the model (prune unnecessary kernels or regularize weights).
  2. Grow the denominator N (feed the model more effective samples).

In computer vision, collecting ten times more labeled medical scans or street photos is expensive. So in 2012, AlexNet popularized Data Augmentation: take each training image and randomly crop it, mirror it horizontally, and slightly jitter its color channels. A cat facing left and the same cat flipped to face right teach the kernel twice as many edge samples without hiring a single human labeler. That increases effective N and pushes the 2σ² / N validation gap back down.

Ten years later, large language model labs learned the exact same lesson at trillion-dollar scale. Between 2020 and early 2022, labs raced to build models with more and more parameters (GPT-3 at 175B, Gopher at 280B, Megatron-Turing at 530B) while training them all on roughly the same ~300 billion tokens of text.

Then in 2022, Hoffmann et al. at DeepMind published the Chinchilla paper. They trained over 400 models of varying sizes and proved that almost every giant model in the world was severely under-trained for its parameter count. Their 70-billion-parameter model (Chinchilla), trained on 1.4 trillion tokens (4× more data, 4× fewer parameters), beat 280B Gopher across the board. Parameters without data just buy expensive estimation error.

BALANCE THE RATIO // PLAY ON ANDROID & iOS

Publish the Overfitting & Chinchilla papers

In Epoch Idle, hitting the validation gap and curing it with Data Augmentation is a rite of passage in Epoch 12. Play free offline with zero forced ads.