"MPLP: Learning a Message Passing Learning Protocol"

[dmm]

I have been looking at a recent rather remarkable paper which includes the DeepDream creator among its authors, and I've decided to check whether I missed any of his works; and I turns out there is this paper I really should be aware of. This really resonates with some of the thing I have been exploring this year.


arxiv.org/abs/2007.00970

"We present a novel method for learning the weights of an artificial neural network - a Message Passing Learning Protocol (MPLP). In MPLP, we abstract every operations occurring in ANNs as independent agents. Each agent is responsible for ingesting incoming multidimensional messages from other agents, updating its internal state, and generating multidimensional messages to be passed on to neighbouring agents. We demonstrate the viability of MPLP as opposed to traditional gradient-based approaches on simple feed-forward neural networks, and present a framework capable of generalizing to non-traditional neural network architectures. MPLP is meta learned using end-to-end gradient-based meta-optimisation. We further discuss the observed properties of MPLP and hypothesize its applicability on various fields of deep learning."

Comments

[dmm]

The one I have been looking at is "Transformers learn in-context by gradient descent"

https://twitter.com/ComputingByArts/status/1605290887013191697

https://arxiv.org/abs/2212.07677

[dmm]

And another interesting recent paper is "Editing Models with Task Arithmetic"

https://twitter.com/ComputingByArts/status/1605264452433092632

https://arxiv.org/abs/2212.04089

[dmm]

"While this paper limits the explorations on this framework to traditional feedforward neural networks, this framework can be used on any graph of agents, as long as their communication protocol remains end-to-end differentiable. We briefly discuss some possible applications of MPLP on non-traditional neural networks in Section 5. The code for this framework and reproduction of the following experiments can be found on

https://github.com/google-research/self-organising-systems/tree/master/mplp"

[dmm]

I need to meditate on how this paper relates to what I have in mind...

This quotation from Section 5 does point out to DMMs as a promising approach in this sense:

"Overall, for few-shot learning, a properly normalized MPLP appears to be a promising path to pursue. All the
experiments explored suggest that meta-learning MPLP for few-shot tasks should work for small-sized networks. If we wanted to scale this up to larger sized networks, this algorithm can quickly become overly expensive. For instance, some of our early experiments implemented convolutions for stateless learners, and we saw how that can very quickly become too computationally expensive. We therefore hypothesize a scaling up might require some significant architectural enhancement, or, more likely, finding a compromise between locality and scalability. One other viable path could be rethinking traditional NN blocks, making them inherently more powerful through MPLP, decreasing architectural depth."