Transformer

Definitions

1. A neural network architecture that uses self-attention mechanisms to process sequential data in parallel rather than sequentially. Introduced in the 2017 paper 'Attention Is All You Need,' it's the 'T' in GPT and the foundation of virtually all modern language models.

   In plain English: The breakthrough AI architecture behind modern language models — it lets the model look at all words in a sentence simultaneously to understand context, rather than reading one word at a time.

Etymology

2017

Vaswani et al. publish 'Attention Is All You Need' — the most consequential 11 pages in recent AI history

2018

BERT and GPT demonstrate transformers' power for language tasks

Related Terms

• Large Language Model
• GPT
• Diffusion Model
• Context Window
• Attention Mechanism
• Mixture of Experts
• Tokenizer
• attention head
• Von Neumann Bottleneck
• Multi-Head