Next Token Prediction

Definitions

1. The task of predicting the next token in a sequence given the preceding context, which is a core training objective for many language models. Despite its simple formulation, next-token prediction can yield surprisingly broad capabilities.

   In plain English: Predicting the next piece of text based on what came before.

   Example: "The paper argued that next token prediction alone was sufficient to produce useful reasoning-like behavior at scale."

Related Terms

• Noise Injection
• Objective Function
• Preference Data
• Refusal Training
• Reward Signal
• Self-Supervised
• Small Language Model
• Training Curriculum
• Training Distribution
• Training Dynamics
• Training Recipe