Alex Mordvintsev (known for DeepDream and more recently for beautiful Neural Cellular Automata and Self-Organizing Textures) created this cool tutorial:
google-research.github.io/self-organising-systems/2022/diff-fsm/"how differentiable optimization can be used to learn Finite State Machines (FSM) for solving toy string processing tasks"
"how simple regularization and initialization techniques can steer continuous optimization towards finding discrete deterministic solutions"
"experiments shown here may have some educational value, e.g. in demonstrating less conventional (and perhaps unexpected) uses of differentiable programming and some
elegant JAX tricks."
He introduces two techniques to "sparsify" the system (to reduce the size of the state machine):
penalty for entropy and addition of identity transform.
:-) I quote-retweeted a summary of this, and Alex retweeted my tweet (I felt honored by that), and that tweet of mine went "semi-viral" as a result :-)
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
