Tokenization Glossary

Browse 10 tokenization terms defined in plain English, from the cultural dictionary of computing.

10 Tokenization Terms

Anchor Token
A token or phrase in a prompt that strongly steers model behavior because it anchors interpretation, format, or topic.
Byte Pair Encoding
A tokenization algorithm that repeatedly merges common symbol pairs to build a useful subword vocabulary. It influences how models are trained, evaluated, or...
Image Tokenizer
A mechanism that transforms an image into a sequence of tokens or discrete units that a model can process, often as part of a multimodal architecture. Image...
Model Tokenizer
The tokenizer associated with a model that converts raw input into tokens and often converts generated tokens back into text. The tokenizer affects context...
Sub-Word
A piece of a word used as a tokenization unit so models can handle rare words, variations, and new forms more efficiently than with whole-word vocabularies...
Token
The basic unit of text processed by a language model — roughly ¾ of a word on average. 'Unbelievable' might be two tokens: 'un' and 'believable.' Token limits...
Token Healing
Techniques used to smooth awkward token-boundary behavior, especially when continuing partially written text or handling boundaries that would otherwise...
Token Merging
A tokenization or model-efficiency technique in which tokens or token-like units are merged to reduce computation or represent text more compactly. The term...
Vocabulary Size
The number of unique tokens available in a model's vocabulary. Vocabulary size affects tokenization efficiency, memory usage, and how text in different...
Word Piece
A tokenization unit smaller than a whole word but larger than a character, commonly used to represent text efficiently in language models. Word pieces help...

Related Topics