Let's understand Large Language Models better

[dmm]

This is a good starting point:

"A Mathematical Framework for Transformer Circuits", Dec 2021
transformer-circuits.pub/2021/framework/index.html

Comments

[dmm]

53:50 Transformers figure out how to clean-up unnecessary leftovers from residual stream

[dmm]

56:00 he speculates that embedding and unembedding only using a fraction of residual stream dimensionality