Understanding the Transformer Architecture

The 2017 paper 'Attention Is All You Need' introduced an architecture that replaced RNNs and LSTMs, reshaped NLP, and went on to revolutionize computer vision, protein folding, and more. Here is how every component of the transformer works, from scaled dot-product attention to mixture-of-experts feed-forward layers.

The Paper That Changed Everything

In June 2017, a team of eight researchers at Google published a paper titled "Attention Is All You Need." The title was deliberately provocative. At the time, the dominant architectures for processing sequences of data, recurrent neural networks (RNNs) and their more sophisticated variant, long short-term memory networks (LSTMs), had reigned unchallenged for years. Every major advance in machine translation, text generation, and speech recognition relied on some form of recurrence. The Google team proposed replacing it entirely with a mechanism called attention.

The result was the transformer, an architecture so effective that within two years it had become the foundation of virtually every state-of-the-art language model. Within five years, it had reshaped not just natural language processing but computer vision, protein folding prediction, music generation, robotics, and weather forecasting.

This article explains the transformer in full: what problem it solves, how each component works, and why it succeeded where other architectures fell short.

The Problem with Recurrence

To understand why transformers matter, you first need to understand the limitations of what they replaced.

Recurrent neural networks process sequences one element at a time, maintaining a hidden state vector that is updated at each step. Given the sentence "The cat sat on the mat," an RNN reads "The," updates its hidden state, reads "cat," updates the state again incorporating information from "The" and "cat," and so on. The hidden state is meant to carry forward all relevant information from every token seen so far.

This design has two fundamental problems that become severe at scale.

The sequential bottleneck. Each step depends on the output of the previous step. You cannot compute the hidden state at position 100 without first computing all states from 1 through 99. This makes training inherently sequential and prevents full utilization of modern parallel hardware like GPUs, which excel at performing many computations simultaneously.

The long-range dependency problem. By the time the model reaches the 500th token, information from the 1st token has been compressed, transformed, and diluted through hundreds of state updates. The mathematical phenomenon known as the vanishing gradient problem makes it extremely difficult for the model to learn relationships between tokens that are far apart. Important context gets washed away.

LSTMs partially addressed the second problem with gating mechanisms, learned functions that control what information to keep, what to forget, and what to output at each step. These gates let information persist across longer distances, and LSTMs became the workhorse of NLP for years. But they could not solve the fundamental parallelization problem. Training remained sequential, and very long-range dependencies (thousands of tokens apart) still degraded.

Attention mechanisms were first introduced as an addition to RNNs, not a replacement. The Bahdanau attention mechanism (2014) let the decoder in a translation model attend directly to all encoder hidden states, rather than relying solely on the final compressed state. This dramatically improved translation quality for long sentences. But the underlying architecture was still recurrent.

The transformer's radical proposition was: what if attention was the entire architecture?

The Core Mechanism: Scaled Dot-Product Attention

The attention mechanism is best understood through an analogy. Imagine you are in a library. You have a question (your Query). Each book on the shelf has a label describing its contents (its Key). You compare your question to each label, identify the most relevant books, and then read the content (Values) of those books, spending more time on the most relevant ones.

Formally, for each token in the sequence, the model computes three vectors through learned linear transformations of the token's embedding:

Query (Q): What information is this token looking for? Key (K): What information does this token offer? Value (V): What content does this token provide when attended to?

The attention score between any two tokens is the dot product of the first token's Query and the second token's Key. These raw scores are divided by the square root of the Key dimension (a scaling factor that prevents the dot products from becoming too large, which would push the softmax into regions where it has extremely small gradients). The scaled scores pass through a softmax function to produce a probability distribution. Finally, these probabilities weight the Value vectors, and the weighted sum becomes the new representation for the Query token.

In notation: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

This single equation is the mathematical heart of the transformer. The rest of the architecture provides structure, stability, and scale around it.

The crucial property: attention is computed between all pairs of tokens simultaneously. There is no sequential processing. Token 1 can directly attend to token 1,000 in a single operation, with no information needing to flow through intermediate states.

Multi-Head Attention: Parallel Perspectives

Language encodes many types of relationships simultaneously. A single word participates in syntactic relationships (subject-verb agreement), semantic relationships (synonymy, analogy), discourse relationships (topic coherence, coreference), and positional relationships (adjacency, clause structure). A single attention computation with one set of Q, K, V projections would be forced to compress all these relationship types into one set of attention weights.

Multi-head attention solves this by running multiple attention computations in parallel, each with independently learned Q, K, and V projection matrices. If the model has h heads and an embedding dimension of d_model, each head operates on vectors of dimension d_model/h. The outputs of all heads are concatenated and projected through a final linear layer.

Empirical research has revealed that different heads learn genuinely distinct functions. In BERT, certain heads specialize in direct syntactic dependencies (one head in layer 8 almost perfectly tracks subject-verb relationships). Other heads implement positional patterns (attending to the previous or next token). Some develop semantic specializations that are harder to categorize but clearly useful for downstream tasks.

The original transformer used 8 heads with a 512-dimensional model. Modern LLMs use 32 to 128 heads with dimensions ranging from 4,096 to 12,288 or beyond.

Positional Encoding: Teaching Order to a Parallel System

Pure attention is permutation-invariant. If you scramble the order of tokens, the attention scores (based on Q-K dot products) do not change because the same pairs of vectors produce the same dot products regardless of their positions. The model would treat "dog bites man" and "man bites dog" identically.

The original transformer addressed this with positional encodings: fixed vectors added to each token's embedding before it enters the model. The paper used sinusoidal functions at different frequencies for each dimension of the embedding:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

This creates a unique positional signature for each position, and the sinusoidal structure ensures that relative positions can be determined from the embeddings (the offset between any two positions produces a consistent transformation).

Later models explored alternatives. Learned positional embeddings (training a separate embedding vector for each position) are simpler but fix the maximum sequence length at training time. Relative positional encodings like ALiBi (Attention with Linear Biases) add a position-dependent bias directly to attention scores, penalizing distant token pairs. Rotary Position Embedding (RoPE) encodes relative distances by rotating the Query and Key vectors in pairs of dimensions, which has become the dominant approach in modern LLMs because it generalizes well to sequence lengths not seen during training and integrates cleanly with the attention computation.

The Encoder-Decoder Architecture

The original transformer was designed for machine translation and used an encoder-decoder structure.

The Encoder Stack

The encoder processes the input sequence (e.g., a sentence in French) and produces a rich contextual representation of it. It consists of N identical layers (N=6 in the original paper), each containing two sub-components:

1. Multi-head self-attention: Every input token attends to every other input token. The word "banque" can attend to "riviere" to determine that this is a river bank, not a financial institution.

2. Position-wise feed-forward network (FFN): A two-layer fully connected network applied to each token independently, with a ReLU (or GELU in later models) activation between the layers. The inner dimension is typically 4x the model dimension, creating an expand-and-compress bottleneck.

Each sub-component is wrapped in a residual connection (the sub-component's input is added to its output) followed by layer normalization. Residual connections create "highways" that allow gradients to flow directly backward through the network during training, preventing the vanishing gradient problem that plagues very deep networks.

The encoder's final output is a sequence of context-enriched vectors, one per input token.

The Decoder Stack

The decoder generates the output sequence (e.g., the English translation) one token at a time. Each decoder layer has three sub-components:

1. Masked multi-head self-attention: The decoder attends to its own previously generated tokens, but with a mask that prevents each position from attending to subsequent positions. When generating the 5th token, the decoder can see tokens 1 through 4 but not tokens 6 onward. This masking is essential: without it, the model could "cheat" during training by looking at future tokens.

2. Multi-head cross-attention: The decoder attends to the encoder's output. The Queries come from the decoder's previous layer, while the Keys and Values come from the encoder. This is the bridge between input and output, allowing the decoder to "look back" at the source sentence while generating each word of the translation.

3. Position-wise feed-forward network: Identical in structure to the encoder's FFN.

Decoder-Only Models

Most modern LLMs (GPT, Claude, Llama, Gemini) use only the decoder half of the architecture. There is no separate encoder and no cross-attention. The input prompt and the generated output are treated as a single sequence. The masked self-attention ensures the model can only see tokens to the left of the current position, maintaining the autoregressive property needed for generation.

This simplification surprised many researchers. The encoder-decoder structure seemed natural for tasks with distinct inputs and outputs (like translation). But decoder-only models proved that a single autoregressive model, trained to predict the next token, could handle translation, question answering, summarization, code generation, and virtually every other language task, all with the same architecture.

Encoder-only models also exist. BERT is the most famous example. It uses only the encoder stack with bidirectional attention (no masking), which makes it excellent for understanding tasks like classification and named entity recognition but unsuitable for text generation.

The Feed-Forward Network: Where Knowledge Lives

The feed-forward network in each transformer layer is sometimes overlooked, but research suggests it plays a fundamentally different role than attention. While attention layers route information between tokens (deciding what information flows where), FFNs appear to store and retrieve factual knowledge.

Studies have shown that specific neurons in FFN layers activate for specific factual associations. One neuron might activate strongly when the model processes text about the Eiffel Tower and Paris. Editing or ablating these neurons changes the model's factual outputs. This has led researchers to describe FFNs as "key-value memories" where the first layer's weights act as keys (matching input patterns) and the second layer's weights act as values (producing the associated output).

Modern architectures sometimes replace the dense FFN with a Mixture of Experts (MoE) layer. Instead of one large FFN, MoE uses many smaller FFNs (the "experts") and a routing network that selects a small subset (typically 1 or 2 out of 8 to 64) for each token. This allows the model to have many more parameters (and thus more knowledge capacity) without proportionally increasing the computation required per token.

Training Stability: The Unsung Heroes

Two architectural choices that seem minor are actually essential to making transformers work at all.

Residual connections add the input of each sub-layer directly to its output: output = x + sublayer(x). In a 96-layer model, the signal must pass through 192 sub-layers (attention + FFN in each layer). Without residual connections, the gradient signal used to update weights during training would either vanish to zero or explode to infinity long before reaching the early layers. Residual connections provide a direct path for gradients, enabling training of arbitrarily deep networks.

Layer normalization standardizes activations to have zero mean and unit variance within each layer. The original transformer applied "Post-LN" (normalizing after the residual addition), but most modern models use "Pre-LN" (normalizing before the sub-layer). Pre-LN produces more stable training dynamics and requires less careful learning rate tuning, which matters enormously when a single training run costs millions of dollars.

Together, these choices are the difference between a model that trains successfully and one that produces numerical garbage.

Why Transformers Won

The transformer's dominance is not due to a single advantage but a combination of properties that align perfectly with modern hardware and data availability.

Massive parallelism. Self-attention processes all tokens simultaneously, fully utilizing GPU and TPU architectures designed for parallel computation. An RNN training on a 1,000-token sequence must execute 1,000 sequential steps. A transformer can process all 1,000 tokens in the same step (though attention is O(n^2) in memory, so there are tradeoffs).

Direct long-range connections. Any token can attend to any other token in a single operation. The "path length" between any two tokens is 1 (one attention step), compared to O(n) for RNNs. This makes learning long-range dependencies fundamentally easier.

Predictable scaling. Transformers exhibit remarkably smooth scaling laws. Performance improves as a predictable power-law function of model size, dataset size, and training compute. This predictability gave organizations the confidence to invest hundreds of millions of dollars in larger models, knowing approximately what improvement to expect.

Modularity and flexibility. The transformer's layer-based design is modular. You can add or remove layers, adjust the number of heads, change the FFN size, or swap components (like replacing dense FFNs with MoE) without redesigning the entire architecture. This flexibility has enabled rapid experimentation and adaptation.

The Quadratic Cost and Efforts to Reduce It

The standard self-attention mechanism computes pairwise interactions between all tokens, giving it O(n^2) complexity in both time and memory with respect to sequence length n. For a 2,048-token sequence, this is manageable. For 100,000 tokens, the attention matrix alone has 10 billion entries.

This quadratic cost has motivated extensive research into efficient attention variants. Sparse attention (attending only to a subset of token pairs, often chosen by locality or learned patterns), linear attention (approximating softmax attention with kernel functions to achieve O(n) complexity), sliding window attention (each token attends only within a fixed-size local window), and FlashAttention (which does not change the mathematical computation but restructures memory access patterns to dramatically reduce GPU memory reads and writes) have all contributed to extending practical context lengths.

These optimizations are why modern models can process 100,000+ token contexts that would have been computationally impossible with naive attention.

Beyond Language: The Transformer Everywhere

The transformer was designed for machine translation, but its architecture is remarkably general. Any problem that can be formulated as operating on a sequence (or set) of tokens can potentially benefit from self-attention.

Vision Transformers (ViT) divide images into 16x16 patches, flatten each patch into a vector, and process the resulting sequence with standard transformer layers. They have matched or exceeded convolutional neural networks on image classification benchmarks.

AlphaFold 2 uses transformer-like attention to model relationships between amino acid residues in protein sequences, enabling the prediction of 3D protein structures with near-experimental accuracy. This was one of the most celebrated scientific breakthroughs of the decade.

Audio and music models apply transformers to spectrograms or discrete audio tokens, producing natural-sounding speech synthesis and music generation.

Multimodal models process text, images, audio, and video through a shared transformer backbone, enabling systems that understand and generate across modalities.

The attention mechanism's ability to model arbitrary pairwise relationships makes it a universal building block. If your data can be represented as a sequence of elements where interactions between elements matter, a transformer can probably learn those interactions.

The Legacy

The "Attention Is All You Need" paper has been cited over 130,000 times. Its eight authors went on to found or co-found multiple AI companies. The architecture they described, with relatively modest modifications, powers systems that hundreds of millions of people interact with daily.

What makes the transformer truly special is not any single clever trick but the convergence of several properties: parallel processing, explicit pairwise token interactions, modular layer design, stable training dynamics, and predictable scaling behavior. From a 100-million-parameter research model to a trillion-parameter production system, the same basic blueprint works.

Few papers in the history of computer science have had such an immediate and comprehensive impact. The transformer did not merely advance the state of the art in natural language processing. It provided a general-purpose computational architecture that is reshaping field after field. Its influence is still expanding.