Token
Noun · AI & Machine Learning
Definitions
The basic unit of text that a language model processes — roughly 3/4 of an English word on average. Text is split into tokens by a tokenizer before being fed to the model. Token count determines cost, context window usage, and processing time.
In plain English: A chunk of text (usually a word or part of a word) that an AI processes as a single unit.
Example: "That prompt is 4,000 tokens — at $0.01 per 1K tokens, each API call costs about 4 cents."
Related Terms
- LLM
- Temperature
- Agentic Workflow
- Guardrails
- LoRA
- Attention
- Prompt Injection
- Tool Use
- Semantic Search
- Beam Search
- BERT
- Byte Pair Encoding
- Causal Language Model
- ELMo
- Function Calling
- Gemma
- GLUE Benchmark
- In-Context Learning
- Large Multimodal Model
- LLM Agent
- LLM Evaluation
- LLM Fine-Tuning
- LLM Inference
- LLM Routing
- LLM Safety
- Local LLM
- Slot Filling