Let's understand Large Language Models better

[dmm]

This is a good starting point:

"A Mathematical Framework for Transformer Circuits", Dec 2021
transformer-circuits.pub/2021/framework/index.html

Comments

[dmm]

They say that compositions often copy sequences (exactly or approximately).

But one should really study the next paper: https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html