Sparsity in Neural Networks (first workshop with this title)

[dmm]

I am very interested in sparsity in neural nets and I am super-happy that the community around sparsity is growing, and the first workshop on the subject had just taken place.

The key organizer is D.C.Mocanu; I did an experimental PyTorch project building upon his work a couple of years ago.

I learned about this via the ML Collective reading group mailing list. The links are in the comments.

Comments

[dmm]

The workshop videos:

https://www.youtube.com/watch?v=pAOzBbMxAmc

https://www.youtube.com/watch?v=fEw2Hw013pk

[dmm]

Workshop on Sparsity in Neural Networks: Advancing Understanding and Practice

Announcement: https://groups.google.com/g/ml-news/c/65k2O2qtzaw?pli=1

The open review page: https://openreview.net/group?id=Sparsity_in_Neural_Networks/2021/Workshop/SNN

The site itself: https://sites.google.com/view/sparsity-workshop-2021/

The site has plenty of informative tabs including https://sites.google.com/view/sparsity-workshop-2021/accepted-papers

[dmm]

The work on this subject I did a couple of years ago, "Regularization in intrinsically sparse networks":

https://github.com/anhinga/synapses/blob/master/regularization.md

https://anhinga-drafts.dreamwidth.org/29924.html

(cross-post: https://anhinga-drafts.livejournal.com/30356.html)

[dmm]

https://www.sparseneural.net/ is the canonical URL

[dmm]

The open tutorial is based on a rather huge survey (given by its first author):

"Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks", https://arxiv.org/abs/2102.00554

So, yes, from an exotic field this topic became something big (although the idea of this standalone workshop still came from a rejection of a workshop proposal by one of the major conferences)

[dmm]

I learned about the new OpenAI Codex paper, https://dmm.dreamwidth.org/44860.html?thread=163132#cmt163132, from the beginning of this tutorial, and I had to switch to studying that paper.

Now this study is done, so can return to focusing on this workshop.