Let's understand Large Language Models better

[dmm]

This is a good starting point:

"A Mathematical Framework for Transformer Circuits", Dec 2021
transformer-circuits.pub/2021/framework/index.html

Comments

[dmm]

Video lecture by Neel Nanda (Oct 2022): https://www.youtube.com/watch?v=KV5gbOmHbjU

via https://twitter.com/NeelNanda5/status/1580782930304978944

[dmm]

"important implications of this residual stream is the sum of paths
29:36
um or model functionality as a sum of paths via the residual stream notion"

[dmm]

"um by projecting it back to the residual stream um one thing to emphasize is I don't
40:32
actually like the words reading and writing here because I think they can be pretty misleading but in particular
40:38
reading and writing intuitively feel like inverses or complementary operations but they're actually very
40:44
different so I prefer the word um project for read and embed for write"

[dmm]

"um side note basically every Transformer I've come across they just hard code the number of
45:34
MLP neurons as four times the residual stream width I don't know why but
45:39
everyone does it so you just memorize the number four"