PART 05 // CONVOLUTION TO TRANSFORMER

When Local Windows Break: From Convolutions to Multi-Head Attention

Convolutions assume neighbors belong together. Attention assumes any token might need to talk to any other token, regardless of distance.

ATTENTION & TRANSFORMERS 7 MIN READ BY LOYAL LOGIC

Every major neural network architecture is a bet about the structure of the data it will see. Researchers call that bet an inductive bias.

The Convolution Bet: Locality and Translation Invariance

When Yann LeCun built LeNet to read handwritten check numbers in the 1990s, he baked two assumptions directly into the wiring of a Convolutional Neural Network (CNN):

  1. Locality (Kernels & Receptive Fields): To tell whether a pixel is part of a vertical edge, you only need to check its immediate 3 × 3 neighborhood. You don't need to look at the opposite corner of the photo.
  2. Translation Invariance (Parameter Sharing): A vertical-edge detector that works in the top-left corner of an image also works in the bottom-right corner. So instead of learning separate weights for every pixel position, you slide one tiny 3 × 3 filter across the whole image at a chosen step size (stride).

That bargain is extraordinary when it holds. A single 3 × 3 RGB kernel has just 27 weights, yet it can scan a 4-megapixel photo. In Epoch Idle, unlocking Conv2D at Epoch 11 lets a small stack of local kernels fit piecewise-segmented curves that a single global neuron can never match.

Where Locality Fails

What happens when the structure of the problem is not local, or when a dataset fractures into more tangled regions than a fixed stack of local windows can tile?

Consider reading a Python file or a legal contract. A variable defined on line 4 is referenced on line 380. If your model only looks through a narrow sliding window, it has no idea what that variable holds. And if you widen the convolution window (Kernel Size) to cover half the page at once, it blurs all the neighboring regions together (in the game, triggering MISALIGNED // PRUNE REQ).

The Attention Bet: Content-Addressed Routing

In 2017, the Transformer paper (Attention Is All You Need) replaced spatial windows with a lookup mechanism borrowed from information retrieval: Queries (Q), Keys (K), and Values (V).

At every layer, each token in the sequence produces three vectors of width d_model:

Attention(Q, K, V) = softmax(Q · Kᵀ / √dₖ) · V Dot-multiply every Query against every Key, divide by √dₖ so large widths don't push the softmax into flat saturation, normalize each row to sum to 1, and take that weighted sum of Values.

Notice that distance does not appear in that equation. Token #1 and Token #2,000 can match with a score of 0.95 on the very first layer if their Query and Key vectors align. And instead of running one giant attention lookup, Multi-Head Attention splits the representation across several independent heads so one head can track syntax, another can track pronouns, and a third can track code indentation in parallel.

IN EPOCH IDLE // EPOCH 20 & THE N×N HEATMAP

At Epoch 20, datasets tangle into 10 distinct regions - past the hard cap of your 4-kernel convolution stack. Only Multi-Head Attention can route extra experts across the remaining regions. Switching the visualizer to ATTN shows the live N × N attention matrix lighting up by head, where you can sweep across cells to focus charge (or automate it with Auto Attend).

The N² Bill and Sparse Mixtures of Experts

Because every token scores itself against every other token, an attention grid for a sequence of length N has N × N = N² cells. Double the context window and you quadruple the attention work. Pushing context windows from 2,048 tokens to 1,000,000 tokens required hardware-aware tiling like FlashAttention and multi-device sharding like Ring Attention.

And once the attention layers were fast enough, the feed-forward weights sitting between them became the new bottleneck. Running a dense 600-billion-parameter network on every single word is overkill when a question about CSS doesn't need the weights that memorized organic chemistry. That triggered the post-Transformer pivot to Sparse Mixture-of-Experts (MoE): keep a stable of 8 or 64 specialist sub-networks, and train a lightweight router to activate only the top 1 or 2 experts per token.

SHIFT THE PARADIGM // FREE ON ANDROID & iOS

Take your model from Conv2D to a Sparse MoE Transformer

Experience the entire progression hands-on in Epoch Idle - free, offline, and built on real equations.