Preprocessing Glossary
Browse 1 preprocessing terms defined in plain English, from the cultural dictionary of computing.
1 Preprocessing Terms
- Tokenizer
- The component that splits text into tokens before feeding it to a language model. Different tokenizers produce different tokens — 'unhappiness' might become...