Why a Straight Line Can't Learn: ReLU, Sigmoid, and GELU
Without a non-linear bend between layers, a 100-layer neural network is algebraically identical to a single line from eighth-grade algebra.
A common question when people first hear that neural networks are just matrix multiplications is: "Why do we need activation functions like ReLU at all? Why not just stack more layers of weights and biases?"
Let's try it with two layers. Layer 1 takes input x and multiplies by weight w₁ plus bias b₁. Layer 2 takes that output and multiplies by w₂ plus b₂:
y = w₂·(w₁·x + b₁) + b₂ = (w₂w₁)·x + (w₂b₁ + b₂) Write
W = w₂w₁ and B = w₂b₁ + b₂. You are right back to y = W·x + B.
It doesn't matter whether you use scalars or 12,000-by-12,000 matrices. The product of two linear transformations is just another linear transformation. If your real-world data curves, thresholds, or saturates, a purely linear network hits a hard mathematical floor and stops improving.
In 1969, Marvin Minsky and Seymour Papert published Perceptrons, proving that a single linear layer couldn't even compute XOR (exclusive-or: output 1 when one input is active, 0 when both are active). That proof froze neural network funding for over a decade. The fix was to insert a non-linear function σ(z) after the linear step so the output could bend.
1. ReLU: The Dumb Hinge That Won Deep Learning
For decades, textbooks taught smooth S-shaped activations. Then in 2010–2012, Nair, Hinton, and Krizhevsky (in AlexNet) popularized the Rectified Linear Unit (ReLU), which looks almost insultingly crude:
z < 0, output 0. If z ≥ 0, output z unchanged.
Why did such a blunt function beat decades of biological neuron curves? Two reasons:
- It costs almost nothing to compute. Comparing a float to zero takes one CPU/GPU instruction; computing exponentials takes dozens.
- Its gradient doesn't decay when active. For any positive input, the slope of
ReLU(z)is exactly1. When you backpropagate error through 50 layers, multiplying by1fifty times still leaves you with1. Your early layers actually receive a learning signal.
2. Sigmoid: When Outputs Must Stay Between 0 and 1
Before ReLU took over hidden layers, the standard activation was the logistic Sigmoid:
(−∞, +∞) smoothly into (0, 1).
Sigmoid is still the right tool when the phenomenon you are modeling saturates at both ends - probabilities, gating switches (like in LSTMs), or S-curved dose-response data. Its weakness in deep stacks is that out at the flat tails (when z is very positive or very negative), the curve goes horizontal. A horizontal curve has a slope near zero, which multiplies incoming gradients by ~0 and starves upstream weights (the vanishing gradient problem).
3. GELU: Why Transformers Dropped the Sharp Corner
Look inside GPT-2, GPT-3, BERT, or ViT, and you won't find plain ReLU. You'll find GELU (Gaussian Error Linear Unit), introduced by Hendrycks and Gimpel in 2016, or its gated cousin SwiGLU:
z by the probability Φ(z) that a standard normal variable is less than z.
ReLU has a sharp corner at z = 0 where the derivative jumps instantaneously from 0 to 1, and for negative inputs it is completely dead (slope 0). GELU rounds off that corner into a smooth dip just below zero before rising linearly. In transformer blocks trained with adaptive optimizers like Adam, that smooth curvature around zero fits soft-hinged patterns that sharp ReLU systematically misses.
In Epoch Idle, unlocking ReLU (Epoch 2), Sigmoid (Epoch 5), and GELU (Epoch 8) doesn't just give a flat "+50% income" buff. Each epoch's dataset is generated from one of those three mathematical families. If you leave ReLU active on a Sigmoid S-curve dataset, your model physically cannot bend at the top tail and stalls at CEILING // σ REQ until you switch.