~36:00 privileged bases in vector spaces come from non-linearities (which is why the residual stream tends not to have any privileged basis)
(but, actually, positions are meaningful, so there is still a bit of privileged structure in the residual stream, just (perhaps) not within the embedding vectors (but perhaps even there, if we look closely, who knows))
~37:50 spectrum of how privileged a basis is, rather than a binary privileged vs non-privileged
(the truth is there are traces of various privileges in the residual stream as well)
~39:30 even ADAM privileges everything it interacts with, because of its weirdness ("ADAM sucks" says Neel Nanda, but I don't think it's necessarily so, perhaps this artificial thing is good, who knows(!)).
no subject
(but, actually, positions are meaningful, so there is still a bit of privileged structure in the residual stream, just (perhaps) not within the embedding vectors (but perhaps even there, if we look closely, who knows))
~37:50 spectrum of how privileged a basis is, rather than a binary privileged vs non-privileged
(the truth is there are traces of various privileges in the residual stream as well)
~39:30 even ADAM privileges everything it interacts with, because of its weirdness ("ADAM sucks" says Neel Nanda, but I don't think it's necessarily so, perhaps this artificial thing is good, who knows(!)).