Unit Glossary

Browse 1 unit terms defined in plain English, from the cultural dictionary of computing.

1 Unit Terms

Token

The basic unit of text processed by a language model — roughly ¾ of a word on average. 'Unbelievable' might be two tokens: 'un' and 'believable.' Token limits...

Related Topics

• Tokenization (1 terms in common)
• Nlp (1 terms in common)