PART 02 // NON-LINEARITY & ACTIVATIONS

Why a Straight Line Can't Learn: ReLU, Sigmoid, and GELU

Without a non-linear bend between layers, a 100-layer neural network is algebraically identical to a single line from eighth-grade algebra.

ACTIVATIONS (σ) 5 MIN READ BY LOYAL LOGIC

A common question when people first hear that neural networks are just matrix multiplications is: "Why do we need activation functions like ReLU at all? Why not just stack more layers of weights and biases?"

Let's try it with two layers. Layer 1 takes input x and multiplies by weight w₁ plus bias b₁. Layer 2 takes that output and multiplies by w₂ plus b₂:

h = w₁·x + b₁
y = w₂·(w₁·x + b₁) + b₂ = (w₂w₁)·x + (w₂b₁ + b₂) Write W = w₂w₁ and B = w₂b₁ + b₂. You are right back to y = W·x + B.

It doesn't matter whether you use scalars or 12,000-by-12,000 matrices. The product of two linear transformations is just another linear transformation. If your real-world data curves, thresholds, or saturates, a purely linear network hits a hard mathematical floor and stops improving.

In 1969, Marvin Minsky and Seymour Papert published Perceptrons, proving that a single linear layer couldn't even compute XOR (exclusive-or: output 1 when one input is active, 0 when both are active). That proof froze neural network funding for over a decade. The fix was to insert a non-linear function σ(z) after the linear step so the output could bend.

1. ReLU: The Dumb Hinge That Won Deep Learning

For decades, textbooks taught smooth S-shaped activations. Then in 2010–2012, Nair, Hinton, and Krizhevsky (in AlexNet) popularized the Rectified Linear Unit (ReLU), which looks almost insultingly crude:

ReLU(z) = max(0, z) If z < 0, output 0. If z ≥ 0, output z unchanged.

Why did such a blunt function beat decades of biological neuron curves? Two reasons:

2. Sigmoid: When Outputs Must Stay Between 0 and 1

Before ReLU took over hidden layers, the standard activation was the logistic Sigmoid:

σ(z) = 1 / (1 + e⁻ᶻ) Squashes any real number from (−∞, +∞) smoothly into (0, 1).

Sigmoid is still the right tool when the phenomenon you are modeling saturates at both ends - probabilities, gating switches (like in LSTMs), or S-curved dose-response data. Its weakness in deep stacks is that out at the flat tails (when z is very positive or very negative), the curve goes horizontal. A horizontal curve has a slope near zero, which multiplies incoming gradients by ~0 and starves upstream weights (the vanishing gradient problem).

3. GELU: Why Transformers Dropped the Sharp Corner

Look inside GPT-2, GPT-3, BERT, or ViT, and you won't find plain ReLU. You'll find GELU (Gaussian Error Linear Unit), introduced by Hendrycks and Gimpel in 2016, or its gated cousin SwiGLU:

GELU(z) = z · Φ(z) ≈ 0.5z · (1 + tanh[√(2/π) · (z + 0.044715z³)]) Weights the input z by the probability Φ(z) that a standard normal variable is less than z.

ReLU has a sharp corner at z = 0 where the derivative jumps instantaneously from 0 to 1, and for negative inputs it is completely dead (slope 0). GELU rounds off that corner into a smooth dip just below zero before rising linearly. In transformer blocks trained with adaptive optimizers like Adam, that smooth curvature around zero fits soft-hinged patterns that sharp ReLU systematically misses.

IN EPOCH IDLE // THE σ LANE

In Epoch Idle, unlocking ReLU (Epoch 2), Sigmoid (Epoch 5), and GELU (Epoch 8) doesn't just give a flat "+50% income" buff. Each epoch's dataset is generated from one of those three mathematical families. If you leave ReLU active on a Sigmoid S-curve dataset, your model physically cannot bend at the top tail and stalls at CEILING // σ REQ until you switch.

HANDS-ON // BEND THE LOSS CURVE

See ReLU, Sigmoid, and GELU fit live data

Download Epoch Idle on iOS or Android to watch the ±σ confidence ribbon bend in real time as you unlock new activations.