Mesa-Optimization

Definitions

1. A concept in AI alignment referring to a learned subsystem that itself behaves like an optimizer pursuing objectives that may differ from the outer training objective. The concern is that such inner optimizers could generalize in unintended ways.

   In plain English: A learned internal optimizer inside a model that may end up pursuing the wrong goal.

   Example: "The paper discussed mesa-optimization as a possible source of surprising behavior when agents encounter situations far outside their training distribution."

Related Terms

• Direct Preference Optimization
• DPO
• Alignment Tax
• Constitutional AI Method
• Expert Iteration
• Goal Misgeneralization
• Goodhart's Law AI
• Gradient-Free Optimization
• Power-Seeking
• Reward Engineering
• Self-Critique
• Specification Gaming
• Surrogate Model