dmm | Entries tagged with scientific papers

You're viewing

dmm's journal
Create a Dreamwidth Account Learn More

Reload page in style: light

I have always been feeling somewhat awkward that my most cited paper has been an industrial pre-machine-learning paper on how to extract geographic references from a text.

Recently, our 2009 paper in American Mathematical Monthly on non-zero self-distances has finally surpassed it in the number of citations.

Meanwhile, my two papers I like the most have exactly zero citations.

5 months in a new job: among software achievements: now I know how to take autocomputed gradients with respect to variables assembled inside nested dictionaries. So I am no longer forced to reshape complicated tree-like-structures into flat arrays in order to use differentiable programming.

As a result, I can finally experiment with DMM training using gradient methods without putting too much labor into those experiments.

🇺🇦 🇺🇦 🇺🇦 Links are in the comments 🇺🇦 🇺🇦 🇺🇦

Alex Mordvintsev (known for DeepDream and more recently for beautiful Neural Cellular Automata and Self-Organizing Textures) created this cool tutorial:

google-research.github.io/self-organising-systems/2022/diff-fsm/

"how differentiable optimization can be used to learn Finite State Machines (FSM) for solving toy string processing tasks"

"how simple regularization and initialization techniques can steer continuous optimization towards finding discrete deterministic solutions"

"experiments shown here may have some educational value, e.g. in demonstrating less conventional (and perhaps unexpected) uses of differentiable programming and some elegant JAX tricks."

He introduces two techniques to "sparsify" the system (to reduce the size of the state machine): penalty for entropy and addition of identity transform.

:-) I quote-retweeted a summary of this, and Alex retweeted my tweet (I felt honored by that), and that tweet of mine went "semi-viral" as a result :-)