Learned optimizers and related topics

[dmm]

github.com/google/learned_optimization - "Meta-learning optimizers and more with JAX"

This is used by various interesting papers including the famous "persistent evolution strategies" paper which I don't understand and "Gradients are Not All You Need" arxiv.org/abs/2111.05803 tempting paper.

Moreover, it is used by a super-interesting "Practical tradeoffs between memory, compute, and performance in learned optimizers" arxiv.org/abs/2203.11860 must-read paper, which is being published at the following conference lifelong-ml.cc/ (Conference on Lifelong Learning Agents - CoLLAs 2022, Aug 18-24)

Comments

[dmm]

Now, the upcoming successor to Zygote.jl is highly categorical and uses Optics all the time, but it is not quite ready yet (no formal releases, not even 0.0.1, although people are still playing with it: https://github.com/JuliaDiff/Diffractor.jl )

Here is how categorical it is: https://gist.github.com/Keno/4a6507b75288b1fe671e9d1cc683014f (no, I don't understand this text)

Apparently, all this is necessary if one wants to handle higher derivatives really well...

The author is https://twitter.com/KenoFischer