Dataflow matrix machines (by Anhinga anhinga)

I have been looking at a recent rather remarkable paper which includes the DeepDream creator among its authors, and I've decided to check whether I missed any of his works; and I turns out there is this paper I really should be aware of. This really resonates with some of the thing I have been exploring this year.


arxiv.org/abs/2007.00970

"We present a novel method for learning the weights of an artificial neural network - a Message Passing Learning Protocol (MPLP). In MPLP, we abstract every operations occurring in ANNs as independent agents. Each agent is responsible for ingesting incoming multidimensional messages from other agents, updating its internal state, and generating multidimensional messages to be passed on to neighbouring agents. We demonstrate the viability of MPLP as opposed to traditional gradient-based approaches on simple feed-forward neural networks, and present a framework capable of generalizing to non-traditional neural network architectures. MPLP is meta learned using end-to-end gradient-based meta-optimisation. We further discuss the observed properties of MPLP and hypothesize its applicability on various fields of deep learning."

Another important paper from one of François Fleuret's collaborations: arxiv.org/abs/2209.00588

Previous important papers include "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention",arxiv.org/abs/2006.16236 and "Flatten the Curve: Efficiently Training Low-Curvature Neural Networks", arxiv.org/abs/2206.07144

Андрей Карпати воспроизводит и исследует первую convolutional neural net (1989):

karpathy.github.io/2022/03/14/lecun1989/

I am very interested in sparsity in neural nets and I am super-happy that the community around sparsity is growing, and the first workshop on the subject had just taken place.

The key organizer is D.C.Mocanu; I did an experimental PyTorch project building upon his work a couple of years ago.

I learned about this via the ML Collective reading group mailing list. The links are in the comments.