6 months since GPT-4 release

[dmm]

A good way to mark this occasion is to try to read a new paper which seems to be a major breakthrough in understanding and harnessing the magic of Transformers:

"Uncovering mesa-optimization algorithms in Transformers"

"we demonstrate that minimizing a generic autoregressive loss gives rise to a subsidiary gradient-based optimization algorithm running inside the forward pass of a Transformer. This phenomenon has been recently termed mesa-optimization"

 

"Moreover, we find that the resulting mesa-optimization algorithms exhibit in-context few-shot learning capabilities,
independently of model scale. Our results therefore complement previous reports characterizing the
emergence of few-shot learning in large-scale LLMs"

 

Comments

[dmm]

https://chat.openai.com/share/674d5916-1d15-47e6-bee0-ea6b0c16113b

ME: There is this definition of "mesa-optimization": "Mesa-Optimization is the situation that occurs when a learned model (such as a neural network) is itself an optimizer. In this situation, a base optimizer creates a second optimizer, called a mesa-optimizer." I wonder what is the etymology of "mesa" here...

GPT-4: The term "mesa-optimization" appears to be derived from the concept of a "mesa," which is a flat-topped elevation or hill with steep sides. The use of "mesa" in the term "mesa-optimizer" can be understood metaphorically to suggest a "higher level" or "platform" atop the foundational structure of a base optimizer. In this context, the "mesa-optimizer" is a model that is trained by the base optimizer and then acts as an optimizer itself, in essence becoming a higher-level system perched on top of the base optimizer.

The terminology comes from the field of machine learning safety research and aims to conceptualize the issues that arise when an optimizer creates another optimizer. One of the concerns with mesa-optimization is that the objectives of the mesa-optimizer may not align perfectly with those of the base optimizer, leading to unintended and potentially unsafe behavior.

The choice of "mesa-" as a prefix likely serves to evoke the idea of layered or hierarchical structures, suggesting a relationship between the base and mesa-optimizer that reflects this structural elevation or nested architecture.