Let's understand Large Language Models better

[dmm]

This is a good starting point:

"A Mathematical Framework for Transformer Circuits", Dec 2021
transformer-circuits.pub/2021/framework/index.html

Comments

[dmm]

MOVING TOWARDS TWO-LAYER MODELS

ATTENTION: The correction (bug fix) in calculation of compositions is a relatively recent addition: according to the Wayback Machine this correction has been added between May 21 and May 24, 2023

[dmm]

IN PARTICULAR, that bug fix happened many months after Neel Nanda recorded his lecture, so expect this error to persist in his presentation.

An interactive interface: https://transformer-circuits.pub/2021/framework/2L_HP_normal.html

[dmm]

They say that compositions often copy sequences (exactly or approximately).

But one should really study the next paper: https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html