How Large Language Models Actually Work

LLMs feel like magic, but they are engineering artifacts built on tokenization, embeddings, and the transformer architecture. This deep dive walks through the complete pipeline from raw text to generated response, explaining every step with clarity and precision.

The Black Box That Isn't

Large language models feel like magic. You type a question, and out comes a coherent, nuanced answer that seems to understand context, humor, and even subtle implication. But LLMs are not magic. They are engineering artifacts built on decades of research in statistics, linguistics, and linear algebra. Understanding how they work does not diminish their impressiveness. If anything, it makes them more remarkable.

This article walks through the complete pipeline: from raw text to trained model to the moment it generates a response. No handwaving, no mysticism, just the actual mechanics.

Tokenization: Breaking Language Into Pieces

Before a language model can process text, it needs to convert human-readable words into numbers. This conversion process is called tokenization, and it is the very first step in the pipeline.

Early approaches used whole words as tokens, but this creates serious problems. English alone has hundreds of thousands of words, and new ones appear constantly. Misspellings, compound words, technical jargon, and words from other languages would all be unknown to a word-level tokenizer. You would need an impossibly large vocabulary, and the model would still encounter words it had never seen.

Modern LLMs use subword tokenization, most commonly Byte Pair Encoding (BPE) or SentencePiece. The idea is elegant: start with individual characters as your base vocabulary, then iteratively merge the most frequently co-occurring pairs. After thousands of merge operations, you end up with a vocabulary that includes common words as single tokens and breaks rare words into recognizable pieces. The word "understanding" might become ["under", "stand", "ing"]. Common words like "the" stay whole. Rare words get split into smaller recognizable fragments.

This approach hits a sweet spot. A vocabulary of 50,000 to 100,000 tokens can represent virtually any text in any language. The model never encounters a truly unknown word because it can always fall back to smaller pieces, even individual bytes if necessary. This is why you can feed an LLM text in Swahili, Python code, mathematical notation, or a mix of all three, and it can process every character.

Each token gets assigned an integer ID from the vocabulary. The sentence "The cat sat" might become [464, 3797, 3332]. These IDs are what the model actually processes from this point forward. The original text is gone; only the token sequence remains.

One important consequence: tokenization is not perfectly word-aligned. A single word might become multiple tokens, and token boundaries can fall in unintuitive places. This is why LLMs sometimes struggle with tasks like counting letters in a word. The model does not "see" individual letters; it sees tokens that may span multiple characters.

Embeddings: From Numbers to Meaning

Token IDs are arbitrary integers. The number 464 does not inherently tell the model anything about the word "the." To make these numbers useful, the model needs to convert them into something that carries semantic information. This is where embeddings come in.

An embedding is a vector, essentially a list of numbers (typically 768 to 12,288 of them, depending on model size) that represents a token in a high-dimensional space. Think of it as coordinates, but instead of the three dimensions we can visualize (x, y, z), you have thousands of dimensions.

The key insight is that these vectors are learned during training. The model discovers that words with similar meanings should have similar vectors. "King" and "queen" end up near each other in this space. "Cat" and "dog" cluster together. "Python" (the snake) and "Python" (the language) might start at the same point but get separated by later layers as the model incorporates context.

The famous example from the earlier Word2Vec era illustrates the power of this representation: the vector for "king" minus "man" plus "woman" produces a vector close to "queen." The model learned gender relationships and royal titles as geometric relationships in vector space, without anyone explicitly teaching it those concepts.

But embeddings capture far more than synonyms and analogies. They encode syntax, sentiment, formality, domain specificity, and dozens of other linguistic properties, all compressed into a single vector per token. These initial embeddings are just the starting point. As the token representations pass through the model's layers, they become increasingly context-dependent and nuanced.

Positional Encoding: Where Are We in the Sequence?

There is a subtlety that needs addressing before the main processing begins. The transformer architecture (which we will discuss next) processes all tokens in parallel. Unlike a human reading left to right, the model sees every token simultaneously. But word order obviously matters: "dog bites man" and "man bites dog" have very different meanings.

Positional encodings solve this by injecting information about each token's position into its embedding. The original approach used sinusoidal functions at different frequencies. More modern models use learned positional embeddings or Rotary Position Embedding (RoPE), which encodes relative distances between tokens rather than absolute positions. The specific method varies, but the purpose is the same: tell the model where each token sits in the sequence.

The Transformer: Attention Is All You Need

The transformer architecture, introduced in a landmark 2017 paper by Vaswani et al., is the engine that powers every modern LLM. Its core innovation is the self-attention mechanism, and understanding attention is the key to understanding everything else.

Self-Attention: How Words Talk to Each Other

Consider the sentence: "The animal didn't cross the street because it was too tired."

What does "it" refer to? The animal or the street? Humans resolve this instantly using context. For a model to do the same, every token needs to "look at" every other token and decide which ones are relevant to its meaning in this particular context.

Self-attention does exactly this. For each token in the sequence, the model computes three vectors from its embedding using learned linear projections:

A Query vector: "What information am I looking for?" A Key vector: "What information do I contain?" A Value vector: "What information do I provide when someone attends to me?"

The model then compares each token's Query against every other token's Key using a dot product. High dot product scores mean high relevance. These scores are scaled (divided by the square root of the key dimension to prevent numerically large values) and passed through a softmax function to produce a probability distribution.

These probabilities become weights that determine how much each token's Value contributes to the output representation. For "it" in the sentence above, the attention weights for "animal" would be much higher than for "street," because the context of "tired" makes the animal the logical referent. The model learns these patterns from billions of training examples.

The result is a new representation for each token that incorporates information from every other token in the sequence, weighted by relevance. The word "bank" gets a different representation in "river bank" versus "bank account" because it attends to different context words in each case.

Multi-Head Attention

A single attention computation captures one type of relationship. But language has many simultaneous relationships: syntactic structure, semantic similarity, coreference, temporal ordering, and more. Multi-head attention runs several independent attention computations in parallel, each with its own learned Query, Key, and Value transformations.

One head might learn to track subject-verb agreement. Another might focus on coreference resolution (connecting pronouns to their referents). A third might attend to nearby words for local phrase structure. The outputs of all heads are concatenated and projected through a linear layer to produce the final output.

Typical models use 12 to 96 attention heads per layer, and they stack 12 to 96 layers deep. The total number of layers times heads gives the model an enormous capacity for representing different types of relationships simultaneously.

Feed-Forward Networks and Residual Connections

Between attention layers, each token's representation passes through a feed-forward neural network. This is a dense, fully connected network that transforms each token independently (no cross-token interaction). Research suggests that these feed-forward layers function as knowledge stores, encoding factual associations learned during training. Specific neurons activate for specific facts, and editing these neurons can change the model's factual outputs.

Residual connections (adding the input of each sub-layer to its output) and layer normalization keep the signal stable as it flows through dozens of layers. Without these architectural choices, the model would suffer from vanishing or exploding gradients and simply fail to train. They are not glamorous but they are essential.

Training: Learning from the Internet

Pre-Training: Predicting the Next Token

The fundamental training objective for decoder-only models like GPT is deceptively simple: given a sequence of tokens, predict the next one. This is called causal language modeling or autoregressive pre-training.

The model reads trillions of tokens from books, websites, code repositories, academic papers, and conversations. For each position in the text, it outputs a probability distribution over the entire vocabulary for what the next token should be, and receives a loss signal based on how wrong it was. The loss function (cross-entropy) penalizes confident wrong answers more than uncertain ones. Over billions of iterations across trillions of tokens, the model adjusts its hundreds of billions of parameters to minimize this loss.

This simple objective forces the model to learn an astonishing breadth of knowledge. To predict the next word in a chemistry textbook, it must learn chemistry. To predict code completions, it must learn programming logic and syntax. To predict dialogue, it must learn social dynamics and conversational patterns. To predict historical text, it must learn history. The prediction task is simple, but solving it well requires deep, broad knowledge of the world as reflected in text.

The scale is staggering. GPT-3 was trained on roughly 300 billion tokens with 175 billion parameters. More recent models train on trillions of tokens with parameter counts in the hundreds of billions or beyond. Training runs cost tens of millions of dollars in compute and take weeks or months on clusters of thousands of GPUs running in parallel.

Fine-Tuning: From Text Predictor to Useful Assistant

A raw pre-trained model is a powerful text predictor but not a useful assistant. It will continue any prompt with plausible text, but it does not reliably follow instructions, answer questions helpfully, or refuse harmful requests. It might respond to a question with another question, or continue your prompt as if it were the next paragraph of an article rather than providing an answer.

Fine-tuning narrows and shapes the model's behavior. Supervised fine-tuning (SFT) trains the model on curated examples of desired input-output pairs: questions paired with good answers, instructions paired with helpful responses.

The most impactful alignment technique is Reinforcement Learning from Human Feedback (RLHF). Human raters compare multiple model outputs for the same prompt and rank them by quality, helpfulness, and safety. A reward model learns to predict these human preferences. The LLM is then optimized using reinforcement learning (specifically PPO or similar algorithms) to produce outputs that the reward model scores highly.

This is why ChatGPT and similar assistants behave so differently from raw language models, despite sharing the same underlying architecture. RLHF shapes the model's helpfulness, safety boundaries, response style, and willingness to decline inappropriate requests.

More recent techniques like Direct Preference Optimization (DPO) and Constitutional AI offer alternative paths to alignment that may be simpler or more stable than RLHF, but the core principle remains: human preferences guide the model toward useful behavior.

Inference: Generating Text One Token at a Time

When you send a prompt to an LLM, the model processes your entire input through all its layers in parallel. This is called the "prefill" phase, and it produces a rich contextual representation of your input. The model then generates a probability distribution over the vocabulary for the very next token.

It samples from that distribution (we will discuss how sampling works shortly) and appends the chosen token to the sequence. Then the full sequence, original prompt plus the newly generated token, feeds back through the model to produce the next token. This autoregressive loop repeats until the model generates a special end-of-sequence token or reaches a maximum length limit.

This is why generation is slower than input processing. Each new token requires accessing the model's full set of parameters. In practice, a technique called KV caching avoids redundant computation: the Key and Value vectors computed for previous tokens are cached and reused, so only the new token requires a full computation at each step. This optimization is essential for practical inference speeds.

Temperature and Sampling Strategies

The probability distribution over the vocabulary determines how the model chooses the next token. Several parameters control this process.

Temperature scales the logits (raw scores) before the softmax function. A temperature of 0 (or very close to it) makes the model nearly deterministic, always picking the highest-probability token. This produces consistent but often repetitive, generic text. A temperature of 1.0 uses the raw learned probabilities. Higher temperatures flatten the distribution, making unlikely tokens more probable and producing more creative, surprising, or occasionally incoherent output.

Top-p (nucleus) sampling filters the vocabulary to only include the smallest set of tokens whose cumulative probability exceeds a threshold p (typically 0.9 or 0.95). This dynamically adjusts the number of candidate tokens based on the model's confidence. When the model is very confident, only a few tokens pass the filter. When it is uncertain, many tokens are considered.

Top-k sampling is simpler: only consider the k most probable tokens, regardless of their probability mass.

These sampling strategies explain why the same prompt can produce different outputs on different runs, and why tuning temperature and top-p dramatically changes the model's creative range and reliability.

Context Windows: The Memory Limit

A context window is the maximum number of tokens the model can process in a single forward pass. Early GPT models had 2,048 tokens (roughly 1,500 words). Modern models support 128,000 or even 200,000 tokens.

The context window is not just an arbitrary limit. Self-attention computes relationships between every pair of tokens, which means computational cost grows quadratically with sequence length. Doubling the context window quadruples the attention computation's memory and compute requirements.

Various techniques reduce this cost: sparse attention patterns that skip distant token pairs, sliding window attention that limits each token to attending within a local neighborhood, FlashAttention (which restructures the memory access pattern for efficiency without changing the mathematical result), and linear attention approximations. These engineering innovations are why context windows have grown from 2K to 200K tokens in just a few years.

Everything outside the context window is invisible to the model. An LLM has no persistent memory between conversations unless an external system provides it. Each request is independent, with only the tokens in the current context window available for the model to reference.

What LLMs Cannot Do

Understanding the mechanics reveals the limitations clearly. LLMs do not search the internet in real time (unless connected to an external tool that does). They do not reason through novel problems the way a mathematician works through a proof, though they can approximate reasoning patterns they have seen in training data. They do not have beliefs, desires, or understanding in the philosophical sense. They are extraordinarily sophisticated statistical models that have compressed patterns in human-generated text into numerical weights.

They can produce false statements with perfect confidence because their training objective is predicting likely text, not verifying truth. They can generate plausible code that does not compile, citations that do not exist, and authoritative-sounding explanations of topics they have no grounding in. This is not a bug in the design. It is a direct consequence of the training objective: produce text that looks like text humans would write.

The Bigger Picture

LLMs represent a convergence of several breakthroughs: the transformer architecture providing the right computational framework, massive internet-scale datasets providing the training signal, enormous compute budgets making training feasible, and clever alignment techniques like RLHF making the resulting models useful and safe.

The field continues to advance rapidly. Mixture-of-experts architectures increase model capacity without proportionally increasing compute costs. Multimodal models extend the same architecture to process images, audio, and video alongside text. More efficient training and inference techniques push the boundaries of what is practical.

But the fundamentals described here, tokenization, embeddings, self-attention, next-token prediction, and sampling, remain the foundation. Understanding these mechanics makes you a better user of these tools. You know why long prompts cost more, why the model sometimes seems to "forget" earlier context in a long conversation, why it can confidently state falsehoods, and why rephrasing a question can produce dramatically different answers. The black box is not so black after all.