Sub-Word

Definitions

1. A piece of a word used as a tokenization unit so models can handle rare words, variations, and new forms more efficiently than with whole-word vocabularies alone. Sub-word tokenization is common in modern language models.

   In plain English: A smaller-than-word text unit used in tokenization.

   Example: "The tokenizer split the product code into sub-word units so uncommon names could still be represented without adding every variant to the vocabulary."

Related Terms

• AI Language
• Image Tokenizer
• Language Understanding
• Model Tokenizer
• Natural Instruction
• Token Healing
• Token Merging
• Vocabulary Size
• Word Piece