REFERENCE // 37 CONCEPTS · 32 PAPERS

Machine Learning Codex & Paper Archive

Every term and paper below ships inside Epoch Idle. Each entry explains the real math first, where it shows up in frontier labs, and how the game models it. Want longer narrative walk-throughs? Read the Beginner ML Field Guide.

01 // ERA GLOSSARY

Foundation

Tokens

∑TOK

Every chatbot answer is assembled one token at a time, and labs live and die by how fast they can push tokens through a model. Tokens are your model's output and the game's score: the more compute you command and the lower your loss, the more you synthesize. Everything on the Train screen exists to raise this number.

tokens/s ≈ compute · multipliers / L²
IN REAL LABS OpenAI reported serving on the order of 100 billion words per day in 2023 (est.). Each one was assembled the way yours are: token by token, against a loss curve.
IN EPOCH IDLE Primary metric and score. Tokens fund research breakthroughs, prestige tiers, and paradigm shifts.

Compute

FLOPs

Training a real model means doing arithmetic, and enormous amounts of it: one training step on a frontier model is quadrillions of multiply-and-adds. FLOPs (floating-point operations) are the currency you spend to train: parameters, hardware, kernels, heads. Research runs on Tokens instead - knowledge is a different budget. Hardware tiers multiply how fast you generate FLOPs.

IN REAL LABS GPT-3's 2020 run took an estimated 3e23 FLOPs; GPT-4 is estimated near 2e25 (est.). One desktop GPU would need about a million years for that job - real labs split it across tens of thousands of chips.
IN EPOCH IDLE Spent to purchase weights, biases, learning rate, and module layers - every era layer's hardware.

AI history is a hardware story: every capability jump of the last decade rode on a new class of chip. Accelerators (CPUs, GPUs, TPUs, Clusters) provide the raw FLOPs required to train larger architectures and execute gradient updates.

IN REAL LABS AlexNet trained on two gaming GPUs in 2012; GPT-4-class runs reportedly use clusters of tens of thousands of accelerators (est.). A modern TPU pod delivers over 1e18 FLOPs per second.
IN EPOCH IDLE Each hardware tier multiplies FLOPs/s ×10 - the engine under every other purchase. Auto-hardware keeps a payback discipline; pause it (Theories screen) to save for a big unlock.

Epoch

E

A real model never sees its data just once: it makes repeated full passes over the dataset, and each pass is an epoch. Here every dataset is a new challenge: converging it banks a permanent multiplier and serves a fresh, harder one. The wave rhythm of the game is the training loop of a lab.

IN REAL LABS Chinchilla (2022) set the modern recipe at 1.4 trillion tokens of text for a 70 billion parameter model. Frontier LLMs mostly see each page once; smaller models sweep curated data repeatedly.
IN EPOCH IDLE Mastering an epoch automatically advances to the next curriculum level.

Convergence

L ≤ ε

Watch a training curve in any AI paper and you see the same shape: loss falls fast, then flattens. A model converges when its loss drops to the noise floor - it cannot fit the data any better without cheating. The game advances the epoch automatically at this point; nothing is ever hard-locked behind waiting.

L ≤ 1.15 (normalized MSE)
IN REAL LABS GPT-3's published loss chart shows exactly this glide into a floor. Labs stare at that curve for months: flattening is the signal to stop burning compute or move to a new architecture.
IN EPOCH IDLE Automatically triggers an epoch advance, granting a permanent ×2.5 multiplier.

Epoch Bonus

×2.5ᴱ

Skills stack: a model that has mastered many tasks learns the next one faster, an effect the field calls curriculum learning. Every mastered dataset permanently multiplies output ×2.5 - experience compounding across the curriculum. It is the game's soft prestige and the reason a Backprop Step (which resets the epoch ladder) is a real sacrifice.

bonus = 2.5^epochsMastered
IN REAL LABS The idea has a lineage: Bengio's 2009 curriculum-learning paper, and the easy-to-hard data ordering in modern LLM training runs.
IN EPOCH IDLE Compounds a permanent ×2.5 multiplier across all token synthesis.

Weight

w

Inside every AI is a giant list of numbers called weights, and training is nothing more than nudging them until the answers get good. A weight scales its input - it is the slope of the model's response. In the conv era one purchase trains every kernel at once (parameter sharing).

y = w·x + b
IN REAL LABS GPT-3 has 175 billion of these dials. Every fact ChatGPT "knows" is stored nowhere else: learned nudges on numbers like your w.
IN EPOCH IDLE Rotates the model toward the slope targets - one gradient trains every kernel at once. On target, the lane reads CONVERGED and refuses to spend until the next epoch resets it.

Bias

b

A line on a graph needs two freedoms: tilt and offset. Weights are the tilt; the bias is the offset, shifting the model's response up or down independent of the input. Without it, every model is pinned to the origin and can never fit offset data.

y = w·x + b
IN REAL LABS The bias has been part of the neuron model since Rosenblatt's 1958 Perceptron, and it survives into every layer of every modern LLM.
IN EPOCH IDLE Shifts the model toward the intercept targets, shared across every kernel. On target it reads CONVERGED and refuses to spend until the next epoch resets it.

MSE Loss

L(θ)

Training needs one number that says how wrong the model is, and that number is the loss. Mean Squared Error is the average squared distance between predictions and data; squaring punishes big mistakes hard, which is why the visualizer tracks the worst-fit point. The game damps production by 1/L² - low loss is a ×64 surge versus a cold start.

L = mean((predict(x) − y)²) / σ²
IN REAL LABS LLMs minimize a cousin of MSE called cross-entropy, and the curve's shape is the same: GPT-3's paper plots it falling from about 3.4 to under 2. Every answer ChatGPT gives started as a fight to keep that number small.
IN EPOCH IDLE Damps token synthesis by 1/L². Driving loss down yields massive throughput surges.

Every AI you have ever used learned the same way: guess, check how wrong you were, nudge, repeat. That nudge rule has barely changed since the 1950s, and GPT-class models still train this way. The learning rate sets how far each nudge moves your weights; too small crawls, too big overshoots. The game caps each step at 95% of the remaining gap, so purchases speed convergence without ever overshooting the optimum.

θ ← θ − η·∇L
IN REAL LABS GPT-3 trained with a learning rate of 0.00006. At 175 billion parameters even a tiny step can overshoot, so this single number gets more tuning attention than almost any other.
IN EPOCH IDLE Multiplies the gap every w/b purchase closes (base 30% × η, capped at 95%). Past the cap (level ~7+) more η closes nothing per purchase - the lane is done.

A network built from straight lines is still just one straight line, however deep you stack it. The activation function is the bend. An activation bends the model's line into a curve: without non-linearity, a network of any depth collapses to a single straight line. Datasets from E3 on are shaped (hinges, S-curves), and only the matching σ family can fit them.

IN REAL LABS ReLU made deep training practical in AlexNet (2012); GPT-2 and GPT-3 chose GELU; most current LLMs use its SwiGLU variant. The choice changes what a network can express.
IN EPOCH IDLE Introduces non-linear curvature (ReLU, Sigmoid, GELU) required for curved datasets.

Gradient

∇L

To know which way to nudge a weight, the model computes the slope of its own error: the gradient. It is the vector of partial derivatives of the loss - the direction of steepest increase. Training steps the other way. The loss-landscape view draws it as the −∇L arrow rolling downhill; the formula card reports it live.

∇L = ∂L/∂θ
IN REAL LABS Backpropagation, the algorithm that computes every gradient in one sweep, was popularized by the 1986 Rumelhart, Hinton and Williams paper (in your archive). One GPT-3 training step backpropagates through 96 layers, giving all 175 billion weights their own gradient.
IN EPOCH IDLE Indicates the direction of parameter descent. Accumulated into ∇ during Backprop Steps.

Every mechanic in this game was once a paper that changed the field. Milestone discoveries publish these papers, documenting the breakthrough you just earned and the mechanic that taught it to you: from Rosenblatt's 1958 Perceptron to Vaswani's 2017 "Attention Is All You Need", the paper behind every modern LLM.

IN REAL LABS The field moves on papers: AlexNet (2012) and the Transformer (2017) each redirected billions of dollars of research within a year or two of publication.
IN EPOCH IDLE Chronicles your lab's historical achievements and unlocks in-depth ML commentary.
02 // ERA GLOSSARY

Convolution

Your phone recognizes faces because a small pattern-scanner slides across the image, and that scanner is a kernel. A convolution kernel is a small local expert: it only sees its window of the input. Stacking kernels lets the model fit piecewise structure a single global line provably cannot - the moment the conv era opens.

IN REAL LABS Yann LeCun's LeNet read handwritten digits this way in 1998; AlexNet scaled it into the 2012 deep-learning boom. Convolutions still run in every camera app: face detection, night mode, QR scanners.
IN EPOCH IDLE Tiles multiple local experts across the data, enabling the model to fit piecewise curves.

How much of the picture should each scanner see at once? That is the receptive field, and the classic answer is 3x3 pixels. Wider windows allow kernels to blend adjacent features and capture regional patterns; the same neighborhood logic repeats at every layer.

IN REAL LABS VGG (2014) proved small stacked windows beat big ones, and the 3x3 kernel became computer vision's default. It is the same trade-off as an LLM's context window, one layer down.
IN EPOCH IDLE Widens every kernel's window for a parameter-sharing production dividend - and widens the taper that blends neighboring segments, which on segmented data can floor the fit above convergence.

The trick that makes vision cheap: one small set of numbers is reused at every position of the image, instead of learning separate dials everywhere. Kernels share one set of trained parameters - the same gradient trains the whole stack. That is the parameter-sharing thesis behind convolutional networks, and the game pays it as a direct production dividend.

bonus = 1 + 0.25·(K − 1)
IN REAL LABS A 3x3 kernel is 27 numbers sweeping the whole image: parameter count stays flat while coverage grows. That asymmetry is why a face detector fits in kilobytes on your phone.
IN EPOCH IDLE Multiplies global token synthesis based on the total number of shared kernels active.

Stride

s

The scanner does not need to examine every pixel: skipping ahead loses little and halves the work. Stride is how far the kernel slides per step. Longer strides scan the input faster - in-game, a ×1.15 training-throughput multiplier per level.

IN REAL LABS Strided convolutions are how every vision network downsamples an image from megapixels to a handful of feature grids - including the patch embedding that feeds photos to vision transformers.
IN EPOCH IDLE Provides a ×1.15 training-throughput multiplier per level.

The oldest trick for more data without more data: flip, crop, and jitter what you already have. Data augmentation multiplies effective training samples by jittering the ones you have. It is the honest cure for overfitting: more data shrinks the estimation error every extra parameter costs.

bonus = 1 + 0.35·level
IN REAL LABS AlexNet (2012) leaned on mirrored images and random crops. LLM training does the text equivalent with paraphrase, filtering, and dedup passes - the same scaling-law lesson, purchasable.
IN EPOCH IDLE Increases effective training samples per window, suppressing validation error.

Overfitting

VAL GAP

A model can ace its own practice test and still fail the real exam: that is overfitting, memorization instead of learning. A model that memorizes noise fits the training split but fails on held-out data - the validation gap. Every expert (kernel or head) must estimate 2 parameters from its window's samples, costing 2σ² of sampling error each: bigger models need more data. Watch the VAL badge, not the training fit.

excess ≈ experts · 2σ² / (n · augment)
IN REAL LABS This is why labs hold out test sets they never train on: a model judged only on data it has seen always looks better than it is. AlexNet shipped with dropout for exactly this reason.
IN EPOCH IDLE Blocks epoch advance and damps output when validation error exceeds noise thresholds.
03 // ERA GLOSSARY

Attention

Attention

A(Q,K,V)

Reading this sentence, you link "it" back to whatever it refers to, however far away. Attention is the mechanism that lets a model do that: it routes information between any two positions based on content, not distance - long-range links no stack of local windows can express. Heads are independent routing experts; covering tangled regions that outgrow the kernel cap is the attention gate.

softmax(QKᵀ/√d)·V
IN REAL LABS The 2017 paper "Attention Is All You Need" replaced every older sequence trick with it. Every LLM since, GPT included, is a stack of these layers: one layer builds thousands of links per token in parallel, and "multi-head" means several kinds of link-finding at once.
IN EPOCH IDLE Resolves non-local correlation on tangled datasets that exceed convolution capacity.

Every word inside the model is a list of numbers, and the length of that list is the model's width. d_model is the width of each token's representation - how much each expert can express. Wider latents make every head worth its slot: the game pays a ×1.5 dividend per level.

IN REAL LABS GPT-3's tokens live in 12,288-dimensional space carried through 96 layers, where the 2017 original used 512. Width is one of the three dials (width, depth, heads) that scaling laws tune.
IN EPOCH IDLE Grants a ×1.5 production multiplier per level for every attention head.

How much text can the model hold in mind at once? That is the context window, and attention makes its cost grow with the square of its length: every token attends to every other. The game's context lane pays that N² back as a quadratic output bonus - the compounding engine of the attention era.

bonus = (1 + level)²
IN REAL LABS The window went 512 tokens (GPT-1) to 2,048 (GPT-3) to 128,000 and beyond (GPT-4 class). Each jump was an engineering war against that square; million-token windows (Gemini 1.5, est. 2024) exist because tricks like FlashAttention shaved its constant factor.
IN EPOCH IDLE Scales output quadratically with (1 + level)², multiplying all synthesis.

Auto Attend

ATTEND

In production nobody hand-routes a model's focus; scheduling systems do it automatically. Auto Attend dynamically concentrates focus on highest-loss regions and continuously reallocates focus charges across heads without manual gestures.

IN REAL LABS The same instinct drives the FlashAttention line of work: the identical math, rearranged so the hardware services the hot spots first. FlashAttention v2 roughly doubled real-world attention speed without changing a single weight.
IN EPOCH IDLE Automates focus charge assignment on the attention heatmap, keeping heads focused.
04 // ERA GLOSSARY

Prestige

Backprop Step

∇ STEP

A real training run ends one of two ways: it converges, or you cut it and keep what the model learned so far. The game's Tier-1 prestige mirrors the loop: EXECUTE EPOCH STEP converts the current run's progress into Gradients ∇ and re-initializes the forward pass, keeping everything you learned about how to learn.

∇ = f(epoch reached × era multiplier)
IN REAL LABS Restarting with retained learning is not a game abstraction: fine-tuning a released checkpoint (GPT-2 alone seeded thousands of downstream models) is exactly this economy, run by labs every day.
IN EPOCH IDLE Tier-1 Prestige: Resets the epoch ladder in exchange for permanent Gradients ∇ spent in the Optimizer Tree.

Optimizer Tree

SGD→Adam

Everyone uses essentially the same optimizer today, and it took the field seventy years to settle on it. Optimizers upgrade the update rule itself, and this tree is the actual historical route: SGD (1951) → Momentum → RMSProp → Adam (2014). In-game each node automates a lane (weights, bias, η, hardware) and multiplies production: automation is the idle payoff of better optimization.

IN REAL LABS Adam (Kingma and Ba, 2014) is the default trainer of essentially every LLM, GPT-3 included, and its paper is among the most cited in computer science.
IN EPOCH IDLE Unlocks automation (auto-buying weights, biases, activations, hardware) and tree multipliers.

Momentum

γv

Push a ball downhill and it keeps rolling through small dips instead of stopping in the first hollow. Momentum adds velocity to gradient descent the same way: updates accumulate a moving average of past gradients, so descent glides through flat spots and ravines. In-game it accelerates lane automation and multiplies tree output.

v = γ·v + η·g
IN REAL LABS Polyak described the trick in 1964 (it is in your archive), and SGD with momentum is still the trainer of choice in computer vision. The γ = 0.90 on your node panel is the field's standard value.
IN EPOCH IDLE Tree node: adds velocity to parameter automation and multiplies tree output.

RMSProp

RMS

Some weights barely matter and some swing wildly; a smart trainer takes big steps on the quiet ones and small steps on the loud ones. RMSProp adapts each parameter's step to its own gradient scale - a moving average of squared gradients. In-game it unlocks adaptive η: the automation learns your learning rate.

θ ← θ − η·g / √(E[g²] + ε)
IN REAL LABS Geoffrey Hinton introduced RMSProp in a 2012 online lecture, and it spread without ever having a proper paper - a famous quirk. Its core idea survives inside Adam, the optimizer that actually trained GPT-3.
IN EPOCH IDLE Tree node: adaptive learning-rate automation and tree multiplier.

Adam

ADAM

Adam is momentum's memory plus RMSProp's adaptivity, and it is the closest thing machine learning has to a universal engine. As the tree's apex it automates the full lane stack and multiplies synthesis - the payoff for mastering both ancestors (gate: max Momentum and RMSProp).

θ ← θ − η·m̂ / (√v̂ + ε)
IN REAL LABS Kingma and Ba, 2014. GPT-3 trained with Adam at β₁ = 0.9, β₂ = 0.95. When someone says a model "learned from the internet", Adam is the loop that did it.
IN EPOCH IDLE Tree apex: full-lane automation (weights, bias, σ, hardware) and the largest multipliers.

No lab trains without saving: power fails, runs crash, and the checkpoint is what you keep. Checkpointing saves trained weights so a crash (or a fresh experiment) never destroys them - and loading a checkpoint into a new run is transfer learning. The game's ongoing Transfer Learning Vault: FREEZE converts mastered epochs into Model Weights ω spent on bonuses that survive every Backprop Step. Freezing resets nothing, and every new run freezes its own waves - deep epochs pay progressively more, so the vault stays worth visiting forever.

IN REAL LABS Every open-weights release, from Llama to Stable Diffusion, is someone's checkpoint frozen mid-curve and handed to the world. BERT (2018) was a checkpoint the entire field fine-tuned from.
IN EPOCH IDLE Transfer Learning Vault: FREEZE banks every mastered-but-unfrozen epoch as Model Weights ω (deep epochs pay superlinearly more), then ω buys the permanent lanes. It survives all resets - claim it any time, not only before a Backprop Step.

Lion

LION

Lion is Adam's successor that threw most of Adam away: it keeps only the momentum's sign - which way to nudge each parameter - and steps every parameter equally. Fewer moving parts, less memory, and it matched Adam across the training runs that tested it. In the tree it is the Tier-2 sink past the apex: pure multiplier, no new automation lanes (gate: max Adam).

θ ← θ − η·sign(m̂)
IN REAL LABS Chen et al., 2023 ("Symbolic Discovery of Optimization Algorithms"): Lion was found by program search, not by hand, and uses 16% less memory than Adam because it stores no second-moment vector.
IN EPOCH IDLE Post-apex ∇ sink: ×4 production per level, no automation changes. This is where surplus gradients go once Adam is maxed.

Depth is how many layers of thinking sit between input and answer, and depth is where abstractions come from: each additional layer composes lower-level features into higher-level ones. In-game DEPTH permanently multiplies output across all runs, surviving both Backprop and Checkpoint resets.

IN REAL LABS ResNet (2015) made 152 layers trainable and pushed ImageNet error below human level; GPT-3 stacks 96 transformer blocks. Depth is why the field is called deep learning.
IN EPOCH IDLE Permanently multiplies output across all runs, surviving both Backprop and Checkpoint resets.

The field does not advance smoothly; it changes horses. When an architecture plateaus, the community switches paradigms: dense nets → convolutions → transformers. The game's Tier-3 shift fires at the token plateau and permanently reroutes your multiplier - the 2017 moment, earned.

IN REAL LABS Each real pivot reorganized the field within a couple of years: transformers did to CNNs what CNNs had done to hand-crafted features, and every frontier model today is a transformer.
IN EPOCH IDLE Permanently transitions the run from Dense Perceptrons to CNNs and Transformers with massive multipliers.
05 // ERA GLOSSARY

Frontier

A dense model runs every parameter on every token - wasteful when most of what it knows is irrelevant to the moment. A mixture of experts instead keeps a stable of specialized sub-networks and a router that sends each token down a few chosen paths: enormous total knowledge, tiny per-token compute. The paradigm shift hands your lab this exact stable - experts are the next lane.

active = top-k(E) of E total
IN REAL LABS Mixtral 8x7B (2023) routed each token through 2 of 8 experts; DeepSeek-V3 (2024) trained 671B parameters while activating only 37B per token (est.). The GPT-6 generation is widely believed to be sparse mixtures (est. 2026).
IN EPOCH IDLE Post-paradigm lane: buy experts on the Train strip to add sparse capacity to the Transformer.

RLHF

RLHF

A base model predicts text; it does not behave. Reinforcement Learning from Human Feedback is the step between: people rank model answers, a reward model learns those preferences, and the base model is tuned against it until it becomes helpful, honest, and safe. Capability is the pretraining; character is the fine-tune.

IN REAL LABS InstructGPT (2022) turned GPT-3 into ChatGPT this way, and Anthropic built the entire Claude line on alignment training (2022 onward, Constitutional AI among it). Every chat product since is a base model plus preference tuning.
IN EPOCH IDLE Reading-room entry: no mechanic - it is why the models behind the papers talk to you at all.

For thirty years the dial was training compute; a new one appeared: thinking longer at answer time. Chain-of-thought prompts draft intermediate steps, and reasoning models spend seconds to minutes of extra inference compute per answer, scoring far higher on hard problems. The scaling law itself moved: what used to be a training cost is now a runtime dial.

quality ≈ f(compute at answer time)
IN REAL LABS OpenAI's o1 (2024) was the first frontier reasoning model; DeepSeek-R1 (2025) reproduced the recipe with open weights. By 2026 the frontier (GPT-6, Claude Fable 5.1) treats test-time compute as a first-class dial (est.).

Agents

AGT

Models stopped only answering. An agent plans a task, calls tools, writes and runs code, checks its own work, and loops until the job is done - the model acting on the world across many steps instead of one. Your lab already runs one: the Auto-Prune agent realigns the expert stack without a finger on it, and ATTEND reallocates focus charges the same way.

IN REAL LABS 2025 was called the year of agents: coding agents shipped real pull requests, and the 2026 frontier stacks (Claude Opus 5.5, GPT-6, est.) are built around tool use as a native capability.
IN EPOCH IDLE Automation lanes (ATTEND, the Auto-Prune agent) are the game's version of the agent loop.

Text-only was step one. Modern models read images, audio, and video inside the same context: pixels become tokens, and attention routes across all of them uniformly. The game's arc mirrors it - the convolution stack learned to see, attention learned to read, and the Transformer runs both in one model.

IN REAL LABS GPT-4V (2023) put vision in the chat window; by 2026 frontier models (Gemini 4, Claude Fable, est.) natively see, hear, and speak. Vision transformers even reuse the convolution-era trick of patching inputs - your stride lane, one domain over.
06 // READING ROOM

Milestone Papers (32)

In Epoch Idle, papers publish automatically when your run hits the historical condition that forced the field to invent them.

The first time a model's loss reaches the noise floor, gradient descent has done its job: the parameters sit on the hidden truth. Every training run since 1958 has been a variation on this moment. In-game, converging banks the ×2.5 epoch bonus and serves a harder dataset.

UNLOCK CONDITION Publishes when you converge your first dataset.

Minsky and Papert (1969) showed a single line cannot even compute XOR - networks need non-linear activations to bend. The game's E3+ datasets are shaped for exactly this reason: only the matching σ family can converge them.

UNLOCK CONDITION Publishes when you unlock your first activation.

Curriculum learning trains on progressively harder examples - easy tasks first, structure later - and converges faster than random ordering (Bengio et al., 2009). The epoch ladder is exactly that: a seeded curriculum of ever-harder datasets.

UNLOCK CONDITION Publishes at epoch 6.

GELU gates each input by its own Gaussian CDF - a smooth, probabilistic ReLU that became the default inside transformers. The game's soft-hinge datasets fit it and nothing earlier: smoothness is a capability, not a detail.

UNLOCK CONDITION Publishes when you unlock GELU (from epoch 8).

Compute per dollar doubled roughly every two years for half a century - the tide that lifted all of AI before the algorithms caught up. Each hardware tier here is ×10 compute for ×12 cost: the same diminishing-return curve, playable.

UNLOCK CONDITION Publishes at hardware tier 4.

In 2012 AlexNet's convolutional network halved ImageNet's error and the field pivoted to learned spatial features overnight. Unlocking Conv2D here is that pivot: local experts replace the single global line.

UNLOCK CONDITION Publishes when you unlock Conv2D (from epoch 11).

Convolution works because structure is local: edges, textures, and motifs repeat across the input, so small receptive windows find them anywhere. Two kernels are the proof of concept - each owns its slice of the domain.

UNLOCK CONDITION Publishes when your stack grows to 2 kernels.

Deep Stack

PAPER

Depth buys composition: each layer's features become the next layer's inputs, building hierarchies no single layer can express. A full kernel stack tiles the whole domain with experts - the capacity that makes segmented structure fittable.

UNLOCK CONDITION Publishes with a maxed kernel stack.

An oversized model stops learning the pattern and starts memorizing the noise: training loss falls while held-out loss climbs. The game charges the exact OLS price - 2σ² of sampling error per estimated parameter - so an oversized stack stalls at OVERFIT // VAL GAP until data catches up.

UNLOCK CONDITION Publishes when your training fit first diverges from validation.

Jitter, flip, crop: augmenting data multiplies effective sample size without collecting anything new, shrinking estimation error directly. It is the honest cure for overfitting - more data, not more parameters.

UNLOCK CONDITION Publishes on your first AUGMENT purchase.

Rich Sutton's observation: methods that lean on raw compute consistently beat clever hand-engineered ones. Stride here is exactly that trade - buy throughput instead of subtlety, and let the machine out-train the tricks.

UNLOCK CONDITION Publishes at STRIDE level 5.

Big Data Era

PAPER

ImageNet (14M labeled images) mattered less for its labels than for its scale: big data is what lets big models generalize instead of memorize. The dataset cap here is 64 points - the same tension in miniature.

UNLOCK CONDITION Publishes when datasets reach 64 samples.

Vaswani et al. (2017) dropped recurrence entirely: stacked self-attention, scaled dot-product similarity, parallel over the whole sequence. Unlocking Multi-Head Attention is the same bet - global, content-based routing instead of fixed local windows.

UNLOCK CONDITION Publishes when you unlock Multi-Head Attention (from epoch 15).

Attention scores every query against every key, then scales by √d so the softmax stays sharp instead of saturating. Multiple heads each learn their own routing table - coverage no single attention map provides.

UNLOCK CONDITION Publishes at 2 heads.

Scaling Laws

PAPER

Kaplan et al. (2020) found loss falls as a smooth power law in model size, data, and compute - predictably, across nine orders of magnitude. The context lane's quadratic bonus is the game's N² scaling law, compounding on schedule.

UNLOCK CONDITION Publishes at CONTEXT level 3.

Chinchilla

PAPER

Hoffmann et al. (2022) rebalanced the scaling law: most large models were starved of data - for a fixed compute budget, train a smaller model on more tokens. The paper fires exactly when you balance model size (context) against data (augmentation): the era's lesson, materialized.

UNLOCK CONDITION Publishes when augmentation and context both grow together.

Some capabilities appear only past a scale threshold - few-shot learning, chain-of-thought - absent one order of magnitude earlier (Wei et al., 2022). Compound the era's lanes far enough and the multiplier simply takes off.

UNLOCK CONDITION Publishes when the era multiplier passes ×10,000.

Rumelhart, Hinton & Williams (1986) made training deep networks tractable: one backward sweep assigns every parameter its share of the blame. The Backprop Step mirrors it - convert a finished run into gradients, then start the next one knowing everything the last taught you.

UNLOCK CONDITION Publishes on your first Backprop Step.

Momentum

PAPER

Polyak's heavy-ball method adds velocity to gradient descent: steps accumulate direction, gliding through ravines and damping oscillation. In-game it automates parameter training - descent that no longer needs a finger on it.

UNLOCK CONDITION Publishes when Momentum comes online in the optimizer tree.

Adam

PAPER

Kingma & Ba (2014) combined momentum with per-parameter step-size adaptation and got the field's default optimizer. The apex node does the same for you: full training automation plus the tree's largest multiplier.

UNLOCK CONDITION Publishes when the Adam apex compiles (both branches maxed).

Checkpoints make training resumable, and loading a frozen model into a new run - transfer learning - became how modern AI is actually built. FREEZE banks each mastered epoch as Model Weights ω: permanent value no reset can touch.

UNLOCK CONDITION Publishes when you freeze your first checkpoint.

Progress is not smooth - it pivots. When the old architecture plateaus, the field switches paradigms and the curve bends again: perceptrons → backprop → convolution → attention. The Transformer shift is the game's version of 2017: the plateau detected, the multiplier rerouted, the next era opened.

UNLOCK CONDITION Publishes when you initiate the paradigm shift at the token plateau.

Modality stopped being a boundary when vision transformers began eating images as patch sequences: pixels became tokens, and attention routed across them like any other text (GPT-4V, 2023). The game's own stack is the argument - convolution learned to see, attention learned to read, and the paradigm runs both in one model.

UNLOCK CONDITION Publishes when convolution and attention have both come online.

Alignment

PAPER

A base model predicts text; it does not behave. InstructGPT (2022) showed a small model tuned against human preferences outperforms a giant one on usefulness: character is a fine-tune, not a parameter count. The game's version - the paradigm multiplies power only once the data discipline that keeps it honest has been paid for.

UNLOCK CONDITION Publishes after the paradigm shift, with augmentation well grown.

o1 (2024) broke a thirty-year assumption: quality is not only a training-budget function. Drafting intermediate steps - chain of thought - turns runtime compute into accuracy, and DeepSeek-R1 (2025) published the recipe. The context lane is your answer-time dial: every extra level is more room to think before answering.

UNLOCK CONDITION Publishes at CONTEXT level 6.

The 2025 shift was not a bigger model but a busier one: plan, act, check, repeat - the loop that turns a language model into a lab assistant. Your lab runs the same loop on itself: the auto-prune agent realigns the expert stack while you are away, no finger required.

UNLOCK CONDITION Publishes when the Auto-Prune agent runs after the paradigm shift.

Sparse routing changed the scaling math: Mixtral 8x7B (2023) ran each token through 2 of 8 sub-networks, and DeepSeek-V3 (2024) trained 671B parameters while activating only 37B per token. Capacity stopped costing per-token compute - your stable adds output without joining the estimation burden a bigger stack pays.

UNLOCK CONDITION Publishes when the first sparse expert joins the routing stable.

The 2026 frontier is widely believed to be a sparse mixture (est.): GPT-6-class systems route every token through specialists instead of running one dense model. A full stable tiles the routing space the way the kernel stack tiles the domain - the paradigm's lesson, completed.

UNLOCK CONDITION Publishes when the expert stable is full.

Mixtral kept the top-2 router honest; the Switch Transformer (2021) went sparser still - one expert per token - and scaled past a trillion parameters on the budget of a dense tenth. Routing, it turns out, is a curriculum: half a stable already teaches the load balancer most of what a full one knows.

UNLOCK CONDITION Publishes when the expert stable reaches four specialists.

Attention's N² memory wall broke when the sequence itself was sharded: Ring Attention (2023) passes key-value blocks around a ring of devices and overlaps the transfer with compute, pushing context windows toward the million-token scale. Your context lane is the same dial - every level is more rope for the window to grow into.

UNLOCK CONDITION Publishes at CONTEXT level 8.

When the public run of high-quality text was exhausted, the labs turned the models loose on the bottleneck: teacher models wrote the textbooks, the exercises, and the grading keys (phi-1, 2023; Self-Instruct, 2022). A frontier-rated model (Elo 1650+) is itself a curriculum engine - the arena is the syllabus.

UNLOCK CONDITION Publishes when the arena rating reaches 1650.

o1 (2024) did not just think longer - it searched: candidate chains proposed, verified, and pruned until one survived, trading wall-clock for accuracy the weights alone could not reach (AlphaGeometry, 2024, showed the same at olympiad geometry). Past Elo 1800 the rating stops measuring what the model knows and starts measuring what it can find.

UNLOCK CONDITION Publishes when the arena rating reaches 1800.
PLAY THE MATH // FREE ON MOBILE

Train every concept on this page

Epoch Idle turns gradient descent, overfitting laws, and multi-head attention into an offline incremental sim. No forced ads, no accounts.