Token
Noun · AI & Machine Learning
Definitions
The basic unit of text processed by a language model — roughly ¾ of a word on average. 'Unbelievable' might be two tokens: 'un' and 'believable.' Token limits define context windows, and token counts determine your API bill.
In plain English: The pieces that an AI breaks text into for processing — not quite words, not quite syllables. A rough rule: 100 tokens is about 75 words.
Etymology
- 1950s
- Compiler design introduces 'token' for the smallest meaningful unit in source code (identifiers, operators, literals)
- 2017
- Transformer models process text as tokens (sub-word units), making token count a key constraint in AI systems
- 2023
- Token-based pricing for LLM APIs makes 'tokens' the unit of AI cost accounting, familiar even to non-technical users