Tokenizer

Noun · AI & Machine Learning

Definitions

  1. The component that splits text into tokens before feeding it to a language model. Different tokenizers produce different tokens — 'unhappiness' might become ['un', 'happiness'] or ['un', 'hap', 'pi', 'ness']. Tokenization choices affect everything downstream.

    In plain English: The component that chops text into small pieces (tokens) before feeding it to an AI model. Different tokenizers split text differently — 'don't' might become one token or three.

Related Terms