Token

Definitions

In large language models, the basic unit of text the model reads and writes. A tokenizer splits text into pieces drawn from a fixed vocabulary, usually learned with byte-pair encoding or a similar subword method, so common words become a single token while rare words are split into several fragments. For English prose a token averages roughly four characters, or about three quarters of a word, but code, numbers, and many non-English languages tokenize less efficiently and consume more tokens per word. Tokens are the unit behind most practical limits: a model's context window is measured in tokens, generation speed is quoted in tokens per second, and API usage is billed per input and output token. Because every model family has its own tokenizer, the same text yields different token counts on different models. Elsewhere in computing, token also names a lexical unit in a compiler and a credential in authentication.
In plain English: The pieces that an AI breaks text into for processing — not quite words, not quite syllables. A rough rule: 100 tokens is about 75 words.

1950s: Compiler design introduces 'token' for the smallest meaningful unit in source code (identifiers, operators, literals)
2017: Transformer models process text as tokens (sub-word units), making token count a key constraint in AI systems
2023: Token-based pricing for LLM APIs makes 'tokens' the unit of AI cost accounting, familiar even to non-technical users