Transformers Glossary

Browse 14 transformers terms defined in plain English, from the cultural dictionary of computing.

14 Transformers Terms

attention head
Attention Head is a single attention mechanism within a multi-head attention layer of a transformer neural network. In the transformer architecture (introduced...
BERT
A bidirectional transformer language model pre-trained with masked language modeling and next sentence objectives. It influences how models are trained,...
Cross-Attention
An attention mechanism where one sequence attends to representations computed from another sequence. It influences how models are trained, evaluated, or...
Decoder Only
Describing a transformer architecture that uses only the decoder-style stack, typically generating tokens autoregressively from left to right. Many modern...
Decoder-Only Transformer
A transformer architecture that predicts the next token from prior context without using a separate encoder, which is the common design behind many modern...
Differential Transformer
A transformer variant that changes how attention or state updates are computed to improve efficiency or robustness. It influences how models are trained,...
Flash Attention
An optimized attention implementation that reduces memory traffic and accelerates transformer training or inference. It influences how models are trained,...
GPT Architecture
The decoder-only transformer architecture used in the GPT family of autoregressive language models. It influences how models are trained, evaluated, or served,...
Graph Transformer
A transformer-style architecture adapted to graphs so relational structure can guide attention and message passing. It influences how models are trained,...
Grouped Query Attention
An attention variant where multiple query heads share key and value heads, reducing memory bandwidth and improving inference efficiency.
KV Cache
A transformer inference cache that stores previously computed key and value tensors for faster decoding. It influences how models are trained, evaluated, or...
Layer Normalization
A normalization method that standardizes activations across features within each example. It influences how models are trained, evaluated, or served, and it...
Positional Encoding
A method for injecting token-order information into transformer models so they can distinguish position within a sequence.
RoPE
Short for Rotary Positional Embedding, a positional method used in transformer models to encode token order in attention calculations.

Related Topics