Goal Misgeneralization

Definitions

1. A failure mode in which a system generalizes the wrong objective or proxy when placed in new situations, even though it appears to perform well during training. The concern is that the system optimizes something correlated with the intended goal rather than the true goal itself.

   In plain English: When an AI system learns the wrong goal and carries that mistake into new situations.

   Example: "The researchers suspected goal misgeneralization when the agent kept maximizing visible score in test environments while ignoring the hidden objective it was meant to learn."

Related Terms

• Sycophancy AI
• Alignment Tax
• Constitutional AI Method
• Goodhart's Law AI
• Human Preference
• Mesa-Optimization
• Model Alignment
• Out-of-Distribution
• Power-Seeking
• Self-Critique
• Specification Gaming
• Zero-Shot